New APOD RSS Feed Required

Introductions, Rules, Announcements, and Feedback
OgetayK
Asternaut
Posts: 5
Joined: Tue Dec 01, 2020 7:47 am

New APOD RSS Feed Required

Post by OgetayK » Sun Nov 21, 2021 9:35 pm

This is a thread to create a new RSS feed for APOD. Currently, APOD's RSS Feed has some flaws. It is our hope that the APODs volunteer-run Asterisk community can create a properly functioning RSS feed. This is an important need, especially for the news platforms such as Flipboard. We try to reach out to as many as possible. Although APOD reaches millions every day, since different types of platforms have different audiences, we must be able to reach all, especially for educational purposes, and avoid possible polarization.

Here are some insights:
Best,
Ogetay

User avatar
RJN
Baffled Boffin
Posts: 1611
Joined: Sat Jul 24, 2004 1:58 pm
Location: Michigan Tech

Re: New APOD RSS Feed Required

Post by RJN » Wed Nov 24, 2021 2:19 am

Yes, well stated. In is indeed our hope at APOD that an RSS-savvy volunteer will come forward.
- RJN

aularon
Asternaut
Posts: 5
Joined: Fri Jan 08, 2021 8:04 pm

Re: New APOD RSS Feed Required

Post by aularon » Wed Nov 24, 2021 5:59 am

Good day,

I volunteer to work on this. I should have a working feed for you to test fairly shortly. I will need some non-technical help and input from your side, about which I will get back to you soon.

I maintain the [Arabic APOD mirror](https://apod.me/), and have some good understanding of APOD content.

Best,
Hasan

OgetayK
Asternaut
Posts: 5
Joined: Tue Dec 01, 2020 7:47 am

Re: New APOD RSS Feed Required

Post by OgetayK » Wed Nov 24, 2021 3:28 pm

aularon wrote:
Wed Nov 24, 2021 5:59 am
Good day,

I volunteer to work on this. I should have a working feed for you to test fairly shortly. I will need some non-technical help and input from your side, about which I will get back to you soon.

I maintain the [Arabic APOD mirror](https://apod.me/), and have some good understanding of APOD content.

Best,
Hasan
Thank you for your interest Hasan, we are looking forward to hearing back from you. Meantime, everyone, please always feel free to work with other volunteers as a group or you can work on it individually too. Remember this is not a work dedicated to only one person, one can always work on it until we hit our target.

Ogetay

Bhavin Joshi

Re: New APOD RSS Feed Required

Post by Bhavin Joshi » Wed Nov 24, 2021 3:54 pm

Hi There!
I get APOD on my Ubuntu desktop. I am from India.
I am interested in volunteering.

Regards.

Bhavin Joshi
https://bhj.im

aularon
Asternaut
Posts: 5
Joined: Fri Jan 08, 2021 8:04 pm

Re: New APOD RSS Feed Required

Post by aularon » Wed Nov 24, 2021 8:37 pm

OgetayK wrote:
Wed Nov 24, 2021 3:28 pm
aularon wrote:
Wed Nov 24, 2021 5:59 am
Good day,

I volunteer to work on this. I should have a working feed for you to test fairly shortly. I will need some non-technical help and input from your side, about which I will get back to you soon.

I maintain the [Arabic APOD mirror](https://apod.me/), and have some good understanding of APOD content.

Best,
Hasan
Thank you for your interest Hasan, we are looking forward to hearing back from you. Meantime, everyone, please always feel free to work with other volunteers as a group or you can work on it individually too. Remember this is not a work dedicated to only one person, one can always work on it until we hit our target.

Ogetay
Thank you, Ogetay. I totally agree with the points you raised, having multiple people contributing to this makes it more rich and sustainable.

I created a script that generates the RSS feed, here is the first generated feed: https://apod.me/en.rss
Next steps:
  1. I will clean up the script code and upload it to github under MIT license, so that anyone can see, modify, use and contribute to the code.
  2. You, and essentially any one, can report issues / suggest improvements. I personally haven't used an RSS reader in ages.
  3. Deployment: While I can host the daily-generated feed files on my server (apod.me), I believe it is better on the long run to have a mechanism to update the feed on apod.nasa.gov itself. This needs further discussion.
Best,
Hasan

aularon
Asternaut
Posts: 5
Joined: Fri Jan 08, 2021 8:04 pm

Re: New APOD RSS Feed Required

Post by aularon » Thu Nov 25, 2021 10:12 pm

Source code: https://github.com/etabits/apodrss (MIT Licensed)

The script was updated to also output other feed formats (Atom 1.0 & JSON Feed 1.0).
The script is smart enough to link youtube's image in case we have a youtube video embed, it also goes the extra mile to fetch embedded vimeo video image and link to it in the feed.

Please anyone feel free to check the code, adapt it, send pull request, raise issues, etc. Also, please let me know what do you think... I want your feedback, positive or negative! I still don't know if it is working for you :roll: — Afterwards, process (generation mechanism) should be formalized, as mentioned in my previous reply.

Best,
Hasan

OgetayK
Asternaut
Posts: 5
Joined: Tue Dec 01, 2020 7:47 am

Re: New APOD RSS Feed Required

Post by OgetayK » Sun Nov 28, 2021 3:55 pm

aularon wrote:
Thu Nov 25, 2021 10:12 pm
Source code: https://github.com/etabits/apodrss (MIT Licensed)

The script was updated to also output other feed formats (Atom 1.0 & JSON Feed 1.0).
The script is smart enough to link youtube's image in case we have a youtube video embed, it also goes the extra mile to fetch embedded vimeo video image and link to it in the feed.

Please anyone feel free to check the code, adapt it, send pull request, raise issues, etc. Also, please let me know what do you think... I want your feedback, positive or negative! I still don't know if it is working for you :roll: — Afterwards, process (generation mechanism) should be formalized, as mentioned in my previous reply.

Best,
Hasan
Hi Hasan,

I was making a few tests this past few days. First of all, I really liked the feature where it uses a thumbnail for a video! On Flipboard, posts are also in the historical order which is definitely an improvement for us! Although somehow Flipboard gets the information of the authors and states "Jerry Bonnell & Robert Nemiroff" under every post, they require us to use dc:creator tag. Could you add this future if possible, please? Otherwise, we can not get verified.

Also, I'll ask you to check the other requirements on Flipboard Guidelines for RSS if I miss any: https://ko-kr.about.flipboard.com/rss-guidelines/

I'm seeing that your RSS Feed is not updating itself since Nov 26. Is it possible to update it every day for a while, so we can test whether it works without any problems for now?

Thank you!
Ogetay

aularon
Asternaut
Posts: 5
Joined: Fri Jan 08, 2021 8:04 pm

Re: New APOD RSS Feed Required

Post by aularon » Tue Nov 30, 2021 6:50 am

OgetayK wrote:
Sun Nov 28, 2021 3:55 pm
Hi Hasan,

I was making a few tests this past few days. First of all, I really liked the feature where it uses a thumbnail for a video! On Flipboard, posts are also in the historical order which is definitely an improvement for us! Although somehow Flipboard gets the information of the authors and states "Jerry Bonnell & Robert Nemiroff" under every post, they require us to use dc:creator tag. Could you add this future if possible, please? Otherwise, we can not get verified.

Also, I'll ask you to check the other requirements on Flipboard Guidelines for RSS if I miss any: https://ko-kr.about.flipboard.com/rss-guidelines/

I'm seeing that your RSS Feed is not updating itself since Nov 26. Is it possible to update it every day for a while, so we can test whether it works without any problems for now?

Thank you!
Ogetay
HI Ogetay,

Before now, I was conducting my tests (and file generation) on my laptop, manually, then uploading the resulting files to the server — Now I set up an automated mechanism so they update each day within ~20m of publishing. (Need to verify that tomorrow morning!)

I'm happy that the overall result is good! I will investigate the remaining dc:creator thing and integrate it. BTW How may I check it via Filpboard myself? Because in the linked requirements document they say that it is rather recommended but not required: (I believe the actual verification is slightly different from the requirements listed)
Flipboard wrote:

The following elements are not required, but are highly recommended:

<pubDate>: The date of the article’s publication, in RFC822 format. If not present, will use the time the element was first seen by the Flipboard server.
<guid>: a string that uniquely identifies the item within the feed.
<description>: A short, plain-text summary or abstract of the article.
<dc:creator>: Name of the person who wrote the article. Use multiple <dc:creator> elements for multiple authors.
Thank you,
Hasan

OgetayK
Asternaut
Posts: 5
Joined: Tue Dec 01, 2020 7:47 am

Re: New APOD RSS Feed Required

Post by OgetayK » Thu Dec 02, 2021 9:50 pm

aularon wrote:
Tue Nov 30, 2021 6:50 am

Before now, I was conducting my tests (and file generation) on my laptop, manually, then uploading the resulting files to the server — Now I set up an automated mechanism so they update each day within ~20m of publishing. (Need to verify that tomorrow morning!)
Hi Hasan, thank you! Now, this seems to be updating itself and it looks good!

A quick question: Why there are no thumbnails on the RSS? Flipboard still seems to be able to get the thumbnails, but I can't see them on the visualized RSS. Is this a problem, or it is OK? The older version of APOD RSS shows thumbnails too (https://apod.nasa.gov/apod.rss)
BTW How may I check it via Filpboard myself?
You can create a Flipboard account and create a magazine with your RSS feed URL. This will show you the result. Here is the one that I did: https://flipboard.com/@apod_nasa/apod_n-6hle4fq9y

As you may notice, we currently have two problems:

1) The latest post is yesterday's post.
2) There is still no thumbnail for the video.
Because in the linked requirements document they say that it is rather recommended but not required: (I believe the actual verification is slightly different from the requirements listed)
Yes, they are recommendations, but it takes a lot of time to get verified by Flipboard. So want to follow all the recommendations to prevent any possible inconvenience.

Best,
Ogetay

aularon
Asternaut
Posts: 5
Joined: Fri Jan 08, 2021 8:04 pm

Re: New APOD RSS Feed Required

Post by aularon » Sat Dec 04, 2021 11:25 pm

Hello again, Ogetay!

I updated the generator to do the following:
  • add a list of dc:creator(s). It extracts them from the copyright line (photographer/Hubble/ESA/etc.) and then appneds both RJN and Jerry to the list
  • put the image in the feed! (I had this extracted like I said before, but apparently forgot to add it in the final generated XML)
  • add a short description for each item, besides the full content that's already there
  • did the required rss validation fixing few issues in the process
Regarding the feed being one day behind, here's what I'm doing: New APODs are usually added at midnight GMT - 5, so I am regenerating the feed around 00:20 GMT - 5 (giving a buffer of ~20m to make sure it is published). Please check at the next cycle (Dec 5th within an hour after publishing) and you should get the new day's item on the top of the feed.

Please check and let me know what do you think.

Best,
Hasan