php - Cron Jobs Managment for large amount of users -

i have service backupify. downloads data different social media platforms, have 2500 active users, each user script runs gets data facebook , stores them on amazon s3, server ec2 instance on aws.

i have entries in table 900 entries facebook users, there php script runs , gets user database table , backups data facebook , picks next user facebook.

everything fine when having less 1000 users, have more 2500 users problem php script halts, or runs first 100 users , halts, time out etc. running php script fro php -q myscript.php command.

the other problem single user scripts takes 65 seconds reach last user database table may take days, whats best way run parrallel on databse table etc.

please suggest me best way backup large amount of data large amount of users, , should able monitor cron, somehting mangaer.

if right, you've got single cron task users, running @ frequency, trying process data of every user in single shot.

1. did try issuing set_time_limit(0); @ beginning of code?
2. also, if task resource demanding, did consider creating separate cron task every n user (basically mimicking multithreaded behaviour; , utilizing multiple cpu cores of server)?
3. is writing data kind of cache instead of database , having separate task commit cache contents database feasible you?
4. do have opportunity use in-memory data table (that's pretty quick)? you'll need persist db contents disk every , then, price fast db access.
5. can maybe outsource task separate servers disributed service , write cron script load balancer them?
6. also optimizing code might help. example (if you're not doing yet) buffer collected data , commit in single transaction @ end of script execution flow not scattered db recurring i/o blockages.