111 replies to this topic

[eric]

I've just published TubePress 4.1.10 which includes the new HTML cache. You can enable and configure this cache from WP Admin > Settings > TubePress > Cache. This should dramatically speed up TubePress on your site.

 

Give it a try and let us know how it works for you? I'm anxious to hear your feedback. Thanks!

[James Israel]

It works very well. It does take a very long time the first time you click on a page in the pagination bar for this gallery: http://www.humortime...2-funny-videos/-- nearly 40 seconds. After that, it's very fast. I'm wondering, however, is this going to be the case each time the cache clears? It's set to clear the default time, 6 hours.

[eric]

Progress!
 

It does take a very long time the first time you click on a page in the pagination bar for this gallery: http://www.humortime...2-funny-videos/-- nearly 40 seconds. After that, it's very fast. After that, it's very fast. I'm wondering, however, is this going to be the case each time the cache clears? It's set to clear the default time, 6 hours.

 
I'm seeing the same when I make it to an uncached page on your gallery (page 6 for me). Took about 30 seconds to load. Given that TubePress is fetching videos from 25 video sources, I'm not entirely surprised that it takes that long for a cache miss.

 

I can think of three solutions for this, and I would love to hear any other ideas in this regard!

1. Bump the cache lifetime to something higher than 6 hours. This is a partial fix, however, since when the cache expires your users will get hit with the cache miss penalty.
    
2. "Prime" the cache a few times a day by visiting each page of the gallery. You could probably automate this by "pinging" each page of the gallery using something like Pingdom. But in a perfect world TubePress would do this for you. My goal is to have TubePress handle all these ugly details on your behalf so I'd like to plan a feature that does just that. Unfortunately I can't offer a relatively quick turnaround time for such a feature as it's considerably more complex to implement than the cache feature in 4.1.10.

Thoughts? I'd be very interested in coming up with anything creative that mitigates the situation until we come up with an elegant solution. Let me know. And thank you so much for helping us improve TubePress!

[James Israel]

Okay, thanks. Meanwhile, I suppose I should look into trying to reduce the number of sources...

[James Israel]

I'm getting a lot of short server timeouts since installing the new version. Could the new cache be hogging resources enough to cause this?

 

Also, question: Do you know if loading playlists vs user from youtube takes more resources? Or does combining the two? Or is it just a matter of how many total we're fetching?

 

Another thing: I notice that when you go back to page one after going to another page in a gallery, it takes just as long to load as the first time you go to other pages. Then, after that, you can go back and forth and page one will load fast. But shouldn't page one already be cached, since we went their first?

 

Lastly, I'd like to set up cron jobs. How do you call the pages in a gallery, like this?

http://www.humortime...ubepress_page=2

 

When I put that in the url bar, it brings the page up. However, I notice that after doing it that way (pasting the url in), it will go fast back & forth to other pages, but then when I go back to clicking on the page number, it loads very slowly again, as if it's not cached. So, not sure what's going on there.

 

I'm thinking of setting the tubepress html cache for 24 hours, and refreshing all the pages late at night every night. What do you think of that idea?

[eric]

I'm getting a lot of short server timeouts since installing the new version. Could the new cache be hogging resources enough to cause this?

 

Seems unlikely to me. If anything it should prevent any kind of server load as it's just fetching HTML from a file and spitting it back out. Were you able to definitely narrow this down to TubePress? I'll be glad to investigate as we certainly don't want to make things worse with the new cache!

 

Also, question: Do you know if loading playlists vs user from youtube takes more resources? Or does combining the two? Or is it just a matter of how many total we're fetching?

 

Playlists vs. user doesn't really make any difference in terms of performance. It's really a matter of the number of sources that TubePress has to fetch. For each one, it has to go out to YouTube, pull back the data, and process it. Network operations like that are inherently slow, which is why we had the API cache. But as we saw even with the API cache TubePress was doing too much processing.

 

Another thing: I notice that when you go back to page one after going to another page in a gallery, it takes just as long to load as the first time you go to other pages. Then, after that, you can go back and forth and page one will load fast. But shouldn't page one already be cached, since we went their first?

 

Nice catch! That's a bug and I can send you the file to update until the next release of TubePress. Would you mind opening a ticket so I can reply with the file there? Would do so here but it's a Pro feature so we can't share it with the world.

 

Lastly, I'd like to set up cron jobs. How do you call the pages in a gallery, like this?

http://www.humortime...ubepress_page=2

 

When I put that in the url bar, it brings the page up. However, I notice that after doing it that way (pasting the url in), it will go fast back & forth to other pages, but then when I go back to clicking on the page number, it loads very slowly again, as if it's not cached. So, not sure what's going on there.

 

First, add this to your shortcode: galleryId="12345". Otherwise TubePress assigns a random ID and your cron requests won't prime the cache. Then your cron HTTP requests will follow this pattern:

http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options%5BgalleryId%5D=12345&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=1

http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options%5BgalleryId%5D=12345&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=2

http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options%5BgalleryId%5D=12345&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=3

You'll just need to update the very last character, which is of course the page number. Again I'm really sorry that TubePress doesn't do this for you already!

 

I'm thinking of setting the tubepress html cache for 24 hours, and refreshing all the pages late at night every night. What do you think of that idea?

 

That sounds perfect - exactly what I would do.

 

Let me know how the cron stuff works. This is the first time I've done this so it might take an iteration or two until we get it right. Thank you

[James Israel]

Seems unlikely to me. If anything it should prevent any kind of server load as it's just fetching HTML from a file and spitting it back out. Were you able to definitely narrow this down to TubePress? I'll be glad to investigate as we certainly don't want to make things worse with the new cache!

 

I don't mean when it's fetching the cached html, I mean when it's creating the cache. I have not narrowed it down, it just seemed logical, since it started happening right after I installed the new version. I did reduce the number of playlists/users to fetch for each gallery yesterday, and the server errors haven't happened since.

 

First, add this to your shortcode: galleryId="12345".

 

So, you mean assign any random number to each gallery in the short code? Does it need to be five digits?

 

I opened the ticket as you requested.

 

Thanks for the cron job info. I'll set that up and let you know how it works.

[eric]

So, you mean assign any random number to each gallery in the short code? Does it need to be five digits?

 

Yes, a large random number is recommended however you can use pretty much any string that you want. It does not need to be five digits. Here are some docs on that option.

[James Israel]

Okay, just to be sure, will the following work for page 1? It would be for 3am. The 'wget --delete-after' placed before the http address and the '2>&1 /dev/null' placed after is what I've seen for other cron jobs, so I just want to be sure they're a good way to go for this cron job:

0 3 * * * wget --delete-after http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options%5BgalleryId%5D=12345&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=1 2>&1 /dev/null

 

Also, please check this out to be sure it's right. It's for another gallery page, but this one has playlists added, whereas the example you gave was for a page that did not have playlists, only users. So, I want to be sure I appended the playlists correctly. They are towards the end. (Also, I changed the gallery ID to 'funny123', will that work ok?):

http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=smbctheater+%2B+TheBilderbergers+%2B+thelonelyisland+%2B+screenjunkies+%2B+latenight+%2B+applesisters+%2B+CorridorDigital+%2B+JimmyKimmelLive+%2B+whiplashpictures+%2B+reformedwhores+%2B+Hollywoodleek+%2B+Satire+%2B+JustKiddingPranks+%2B+owenweberlive+%2B+DailyFiberFilms&tubepress_options%5BplaylistValue%5D=BPLEE6E4AF38BC19EC2+%2B+PLD7nPL1U-R5qLnFC8fcGKyn4nQAP8LkLs&tubepress_options%5BgalleryId%5D=funny123&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=1

[brandon]

James,

 

Both of those should work as the Gallery ID can be any alphanumeric combination of characters

 

Per the documents: "Valid values  Any string, though a large positive integer is recommended."

 

Thanks!

[James Israel]

Okay, but in the second example, I'm asking about the addition of the youtube playlists to the cron job (vs just youtube users). Can you check it and make sure that part is ok? It's the part after the last user listed (DailyFiberFilms) that starts:

&tubepress_options%5BplaylistValue%5D=

Thanks!

[brandon]

James,

 

That should work fine.

[James Israel]

So, I tested the cron job, and at first I didn't think it was working at all. But then, something strange: a different page was cached than what I put in the cron job. Could be a coincidence, like someone had just looked at that page, and that's why it was cached, I don't know.

 

The cron job I put up was for page 2 of the http://www.humortime...2-funny-videos/gallery. (It ended with &action=tubepress&tubepress_page=2) It loaded slowly today, like usual. Then I tried page 3, and it popped right up.

 

Can you think of an explanation for this, other than the coincidence factor?

 

I have the cache expiration set for 24 hours, 86400 seconds. I also set the 'cleaning factor' to 0. Is that how that should be set? Not sure what that really means.

 

Also, I looked on the server where the other tubepress caches are, and there was no new cache written after the cron job. The latest cache was dated June 19th. In the Tubepress setup for html cache, I left the directory blank, like the api cache, so it should be using the same tmp directory.

 

The full cron job code I submitted (using crontab -e) was:

2 3 * * * wget --delete-after http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options%5Bmode%5D=playlist+%2B+user&tubepress_options%5BresultsPerPage%5D=1&tubepress_options%5BresultCountCap%5D=25&tubepress_options%5BorderBy%5D=newest&tubepress_options%5BperPageSort%5D=newest&tubepress_options%5BuserValue%5D=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options%5BgalleryId%5D=funny001&tubepress_options%5BadjustedResultsPerPage%5D=4&action=tubepress&tubepress_page=2 2>&1 /dev/null

I have galleryId="funny001" in the shortcode on that page.

 

Thanks!

[brandon]

Very strange indeed.  Do you have any cron errors in your *nix root mailbox?  These are usually really good at telling you what went wrong,

 

When I'm working with *nix devices, I always change emails sent to root to forward to an actual email address so I can see the error messages and fix cron errors.

 

Thanks!

[James Israel]

The cron.log does not seem to show any errors. Only a list of the regular cron jobs that trigger the wp-cron.php file:
 

Jul  5 06:30:01 content /USR/SBIN/CRON[18587]: (root) CMD (wget --delete-after http://www.humortimes.com/wp-cron.php2>&1 /dev/null)
Jul  5 06:35:01 content /USR/SBIN/CRON[18606]: (root) CMD (wget --delete-after http://www.humortimes.com/wp-cron.php2>&1 /dev/null)

 

 

 

...and this line, not sure what it means:

Jul  5 06:39:02 content /USR/SBIN/CRON[18614]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

 

 

[James Israel]

Any suggestions on getting this cron job to work? Doesn't seem to have any effect yet. Thanks!

[eric]

Hi James,

 

Sorry for the delay in response. I just visited your page to take a closer look at things and noticed a new problem; clicking on any of the pagination links is badly broken and it basically kills the entire gallery. I spent the past half hour or so investigating and think that it has something to do with CloudFlare's Rocket Loader. Just to test, could you temporarily disable Rocket Loader for your videos page? I would suggest putting in a page rule to do this. Another alternative would be to simply disable Rocket Loader altogether for a short time, just to see if it fixes TubePress.

 

Let's get your gallery back in working order and then we'll tackle the cron issue. Sound good? Thanks!

[James Israel]

Okay, I made the page rule and it seems to be okay - should apply to all our video galleries. I had just turned rocket loader back on after being off for months, due to another issue.

[eric]

Thank you - I can confirm that pagination is working again. I'll make a note to figure out why Rocket Loader breaks TubePress (it shouldn't, in theory).

 

I'll investigate the cron issue tonight and report back what I find. Please stand by, thanks!

[eric]

Just did some investigation and have a few leads. This is the cron job that you have in place currently:

2 3 * * * wget --delete-after http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options[mode]=playlist+%2B+user&tubepress_options[resultsPerPage]=1&tubepress_options[resultCountCap]=25&tubepress_options[orderBy]=newest&tubepress_options[perPageSort]=newest&tubepress_options[userValue]=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options[galleryId]=funny001&tubepress_options[adjustedResultsPerPage]=4&action=tubepress&tubepress_page=2 2>&1 /dev/null

First, wget is particular about query parameters and requires that you enclose the entire URL in double quotes. So one thing to try would be this command instead:

2 3 * * * wget --delete-after "http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options[mode]=playlist+%2B+user&tubepress_options[resultsPerPage]=1&tubepress_options[resultCountCap]=25&tubepress_options[orderBy]=newest&tubepress_options[perPageSort]=newest&tubepress_options[userValue]=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options[galleryId]=funny001&tubepress_options[adjustedResultsPerPage]=4&action=tubepress&tubepress_page=2" 2>&1 /dev/null

That should work, but I haven't tested it. However I did test the following command and confirmed that it successfully primed the cache for page 5:

2 3 * * * curl "http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options[galleryId]=funny001&tubepress_options[mode]=playlist+%2B+user&tubepress_options[resultsPerPage]=1&tubepress_options[resultCountCap]=25&tubepress_options[orderBy]=newest&tubepress_options[perPageSort]=newest&tubepress_options[userValue]=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options[adjustedResultsPerPage]=4&tubepress_page=5&action=tubepress" 2>&1 /dev/null

This uses curl instead (which for our purposes works just as well as wget), and I copied the URL directly from Chrome Dev Tools so the tubepress_page parameter is second to last, instead of last. That shouldn't make a difference but figured I'd keep it identical to what happens in the browser. If it doesn't matter to you, I'd suggest using the curl solution as I've confirmed that it works.

 

Thoughts?