111 replies to this topic

[James Israel]

I notice you are using square [brackets] instead of the %5B type codes this time, is that intentional, or should they be replaced?

 

I don't have curl installed on my server, so I'll just try the wget, but plz let me know on the above.

[eric]

Definitely use square brackets, not %5B. I'm sorry if I had used that erroneously in a previous message. I just tested the following command and it was able to prime page 4 successfully:

wget --delete-after "http://www.humortimes.com/wp-admin/admin-ajax.php?tubepress_action=shortcode&tubepress_options[mode]=playlist+%2B+user&tubepress_options[resultsPerPage]=1&tubepress_options[resultCountCap]=25&tubepress_options[orderBy]=newest&tubepress_options[perPageSort]=newest&tubepress_options[userValue]=BuzzFeedVideo+%2B+klaatu42+%2B+HigtonBros+%2B+MediocreFilms+%2B+avengersweb+%2B+ChessClubComedy+%2B+pleatedvideo+%2B+TheSecondCityNetwork+%2B+Cracked+%2B+ERB+%2B+RhettandLink+%2B+cultcomedypictures+%2B+collegehumor+%2B+broadcomedy+%2B+UCBComedy+%2B+ImprovEverywhere+%2B+HISHEdotcom+%2B+SilvermanVideos&tubepress_options[galleryId]=funny001&tubepress_options[adjustedResultsPerPage]=4&action=tubepress&tubepress_page=4"

[James Israel]

Great, thanks. I had the cron job installed for page 5 last night, and it was working as well today. I've gone ahead and installed a bunch of them, which should be triggered tonite, I'll let you know how they work. For now, I'm doing pages 1-4 for each of the four "Funny Video" galleries and for the "Political Humor Videos" gallery. There are seven or more pages for each, but I figure most people won't go beyond page two or three anyway.

 

Also, regarding a previous issue with going back to page 1 from another page in the pagination, that still takes a long time to load, so I am caching it as well (even though that shouldn't have to be cached separately, it seems to me, since the initial page is cached by tubepress -- something about the way pagination works, I guess).

[eric]

Great, thanks. I had the cron job installed for page 5 last night, and it was working as well today. I've gone ahead and installed a bunch of them, which should be triggered tonite, I'll let you know how they work. For now, I'm doing pages 1-4 for each of the four "Funny Video" galleries and for the "Political Humor Videos" gallery. There are seven or more pages for each, but I figure most people won't go beyond page two or three anyway.

 

That's great to hear! Woohoo! Just to reiterate, my ultimate goal is to have TubePress prime the cache for you. So in the future this shouldn't be necessary.

 

Also, regarding a previous issue with going back to page 1 from another page in the pagination, that still takes a long time to load, so I am caching it as well (even though that shouldn't have to be cached separately, it seems to me, since the initial page is cached by tubepress -- something about the way pagination works, I guess).

 

That issue is still on my plate and I haven't forgotten. I'll update you via the ticket that you submitted.

 

Thanks again for your patience with us!

[James Israel]

Well, bad news: this morning, only page 3 of the Funny Videos - Gallery 1 (http://www.humortime...2-funny-videos/) seemed to be cached. The others loaded very slowly, as before. For all I know, page 3 was only cached because someone checked it out earlier. Same for the other galleries (Funny Videos - Gallery 2, http://www.humortime...unny-videos-2/; etc thru Gallery 4; also, Political Humor Videos' pages, except for page 2 there, which was cached).

 

I have the html cache in Tubepress set for 24 hours, so they should stick if they're working. API cache is only 6 hours, should I increase that to 24?

 

Also, I have the cache directories blank (so it should use the default), however, I can't find any cache directory on the server. Do you know where it should be? The wp-content/cache folder does not seem to have any tubepress cache files in it. Since the API cache seems to be working, it's gotta be somewhere...

 

So the positive result yesterday may have just been the dumb luck of someone visiting the page recently, I don't know. Examining the cron codes, they all look good, but in case you see something I don't, I could post all of them, or email them to you. It's pretty long, what with four pages each for five different galleries.

[James Israel]

Update: I did find the tubepress caches, they are at the server's root /tmp directory. One of them has today's date, the 27th, so presumably, that would be the one. It has both a api-calls subdirectory and one called cache-html. Drilling down thru several subdirectories with names like "57cd99682e939275", I came to one with about 240 subdirectories. Inside one of those, I found a php file, opened it, and the content, which looked like it would serve up a gallery page, was:

 

<?php
/* Cachekey: cache/stash_default/1b437d3af17fec4b08addd675fc8a6ac/ */
/* Type: array */



$loaded = true;
$expiration = 1438048519;

$data = array();

/* Child Type: string */
$data['return'] = "
<div class=\" js-tubepress-gallery js-tubepress-gallery-funny004\">

<div class=\"tubepress-normal-player js-tubepress-player-normal\" style=\"width: 100%\">


                <div class=\"tubepress-big-title\">GN: How Hot Dogs are REALLY Made</div>

<iframe width=\"800\" height=\"480\" src=\"https://www.youtube....load_policy=3\"frameborder=\"0\" class=\"js-tubepress-embed\" webkitallowfullscreen mozallowfullscreen allowfullscreen id=\"tubepress-player-1413700260\" name=\"tubepress-player-1413700260\"></iframe><script type=\"text/javascript\">
   var tubePressDomInjector = tubePressDomInjector || [], tubePressYouTubePlayerApi = tubePressYouTubePlayerApi || [];
       tubePressDomInjector.push(['loadPlayerApiJs']);
       tubePressDomInjector.push(['loadJs', 'web/js/youtube-player-api.js', true ]);
       tubePressYouTubePlayerApi.push(['register', 'xGE1JVmJe-0', 'tubepress-player-1413700260' ]);
</script><script type=\"text/javascript\">
   var tubePressDomInjector = tubePressDomInjector || [],
       tubePressResponsiveEmbeds = tubePressResponsiveEmbeds || [];
       tubePressDomInjector.push(['loadJs', 'http://www.humortime...sive-embeds.js'true ]);
       tubePressResponsiveEmbeds.push(['register', 'tubepress-player-1413700260', {}]);
</script>

</div>

 

[James Israel]

Follow up question: Will all these cache files keep piling up, or will they overwrite each other? I have the html cache cleaning factor set at 0 right now, should I set that to something else?

[eric]

I have the html cache in Tubepress set for 24 hours, so they should stick if they're working. API cache is only 6 hours, should I increase that to 24?

 

The HTML cache takes precedence over the API cache, so you can leave those settings as they are. In other words, if there's a cache hit in the HTML cache, the API cache won't even be considered.

 

So the positive result yesterday may have just been the dumb luck of someone visiting the page recently, I don't know. Examining the cron codes, they all look good, but in case you see something I don't, I could post all of them, or email them to you. It's pretty long, what with four pages each for five different galleries.

 

I was going to ask you if I could see the cron calls, yes. Please submit them via a ticket if you would, just so that anyone else reading this thread isn't drowned in text What I might do is put them into a shell script so that cron can just call the script - beats having to manage a million cron jobs.

 

Update: I did find the tubepress caches, they are at the server's root /tmp directory. One of them has today's date, the 27th, so presumably, that would be the one. It has both a api-calls subdirectory and one called cache-html. Drilling down thru several subdirectories with names like "57cd99682e939275", I came to one with about 240 subdirectories. Inside one of those, I found a php file, opened it, and the content, which looked like it would serve up a gallery page, was ...

 

That is the anticipated structure and behavior of the cache (although 240 subdirectories seems a little high, but not out of the question). The file you sampled looks good.

 

Follow up question: Will all these cache files keep piling up, or will they overwrite each other? I have the html cache cleaning factor set at 0 right now, should I set that to something else?

 

Good question. They will overwrite each other. Basically TubePress computes a cache key based (currently at least) on the URL of the page. So once a page expires, it will be recomputed and then stored in place of the old one.

 

Please send your cron commands when you get a moment. We're close - I can feel it!

[James Israel]

Tried, but your support site was not working well ... extremely slow, and didn't seem to take my first submission, not sure. Will try again later. Btw, I've had trouble with your support site being very slow many times ... might be something you guys need to take a look at.

[eric]

Tried, but your support site was not working well ... extremely slow, and didn't seem to take my first submission, not sure. Will try again later. Btw, I've had trouble with your support site being very slow many times ... might be something you guys need to take a look at.

 

Hmm that's the first report we've had in that regard. We monitor all of our sites (including our support site) with Pingdom and I don't see any unusual performance issues there. That site is backed by desk.com and I do see one or two service interruptions yesterday - not sure if that impacted you or not.

 

If you continue to experience issues with that site, please make a note of which page you were on and shoot us a note. We obviously want to keep things running at peak performance!

 

I got your support ticket that you were able to send through and am taking a look now. Please stand by..

[James Israel]

Any word on this cache thing yet? Thanks.

[eric]

Went to work on this yesterday but ran into a problem; when I visit this page, I see "No matching videos" instead of the gallery. At first I thought TubePress had encountered a problem and was serving the "No matching videos" message from its HTML cache. So I tried to clear the HTML cache by visiting:

 

http://www.humortime...stem_cache=true

 

That seemed to clear the cache as I saw a gallery again, but then when I revisited the original page, I saw the "No matching videos" message again. So I'm wondering if perhaps your WordPress caching plugin (W3TC?) is serving up the stale data. Could you give that a test and maybe clear your WordPress cache to see if that clears it up?

 

I feel like we've been putting you through the ringer over the past few weeks and I sincerely apologize! You are a TubePress power user in many ways and are testing the limits of our software, which in turn is bringing bugs to the surface. I really appreciate your patience with us but also realize that your patience is finite. I feel we are very close to getting things squared away for good.

 

Thanks

[James Israel]

Hey, no problem, I realize I am asking a lot of your software, and that it'll take some time to work out. Glad to help you improve it.

 

I had changed my Youtube API, related to another project (an app), and forgot I had to update it in Tubepress. It's working now.

[eric]

Looks much better now - good stuff. Working on the cron script now...

[eric]

OK, good news. I wrote up a simple Bash script that you can invoke nightly. It will iterate over the first 10 pages for all 5 of your galleries (funny 1-4 and political). Here's how to use it:

• Download the contents of this page to a file named prime-tubepress.sh, or something like that.
   
• Upload prime-tubepress.sh to your server somewhere.
   
• Ensure that the file is executable. On *nix systems this will entail something like chmod a+x prime-tubepress.sh
   
• Get rid of any previous cron entries related to priming TubePress's cache
   
• Add a cron entry like this:
  

  8 3 * * * /path/to/prime-tubepress.sh 2>&1 /dev/null

That's it! The script works very well in my testing, although it takes about 20 - 30 minutes to finish. It's running right now so if you check your galleries in a while they should be fully primed and fast. The only real caveat to this solution is that if you change the shortcodes you use for any of your galleries, you'll need to update the script. If you'd like, I can show you how to grab the URLs that the script uses.

 

As I've said, in the future I want TubePress to handle all of this for you, but for now this should work. Give it a try?

[James Israel]

Just seeing this now ... I've installed as directed. Only thing, there seemed to be a typo in each entry in the prime-tubepress.sh file: In every case, the first playlist listed after the 

&tubepress_options[playlistValue]=

line did not have the 'PL' in front of it. I inserted them -- I hope it wasn't supposed to be the way you had it for some reason. If so, I can go back and replace it.

 

The cron job is schedule for 3am Pacific Time, I'll check it out tomorrow and let you know if it's working.

 

Thank you!

[eric]

Did you get a chance to check your galleries this morning? I just checked one of them now and it didn't seem to work; I clicked on page 6 and it took like 30 seconds to load. I would first try running the script manually on your server (assuming you have SSH access), and then I would remove 2>&1 from the end of the cron line so that you could get a glimpse of any errors.

 

 

Only thing, there seemed to be a typo in each entry in the prime-tubepress.sh file: In every case, the first playlist listed after the 

&tubepress_options[playlistValue]=

line did not have the 'PL' in front of it. I inserted them -- I hope it wasn't supposed to be the way you had it for some reason. If so, I can go back and replace it.

 

YouTube ignores any leading "PL" in a playlist ID, so it won't hurt anything if you inserted them in again.

[James Israel]

Not sure if I'm doing it right, but when I try to run the file via this command:

sh prime-tubepress.sh

 

... I get: "tmp.txt: Permission denied" about 50 times.

 

I set prime-tubepress.sh up as a 'root' owned file, but since the crontab is owned by my account (humortimes) on the server, perhaps I need to have the script owned by humortimes?

[James Israel]

Okay, ran the script again using 'sudo' to get permission, and it ran. After each cron job, it gave a '200 OK' response, then saved a 'tmp.txt' file, then deleted it, which I see is in the script to do.

 

After it ran, I tried some pages, and some seemed to work, others not. The Political Humor Videos page (political001 in the script) seems to work for all the pages in the pagination, but the first Funny Videos page (funny001) did not. For example, I could see in the running log that funny001, pagination page 5 (tubepress_page=5) ran, but when I tried that page, it was still very slow, while all the pages I tried on the Political page were fast.

 

Funny Videos page 2 (funny002), pagination 2 seemed faster than normal, but slower than the Political pages - it took a few seconds to load, but pagination 3 was fast. So, mixed results, it seems.

 

However, the more I tested, the more it seems only the first Funny Videos page (funny001) wasn't working. I went to that page and checked, and it does have the   tubepress galleryId="funny001"   line there. Didn't notice anything odd about the script for that page, but maybe it's something there.

 

Also, when I checked originally, before manually running the script, I'm not sure which pages I checked, but maybe it was only funny001, in which case, maybe the cron job is working for the other pages. I guess I won't know for sure til tomorrow. Still not sure how the ownership for the script should be set.

[eric]

I would suggest that you set ownership of the script to humortimes, not root. If you're getting a "Permission denied" error it means that the script can't write the tmp.txt file. Here's an updated version of the script that tells wget not to save the output at all, which will get around this issue entirely.

 

I just ran the script from my local machine here and it was able to prime all pages of the galleries. So, I'd say let's give this new script a try on your server and with some luck we'll be in business. One way or another, we're close!