NASA Open API for APOD

Introductions, Rules, Announcements, and Feedback
User avatar
geckzilla
Ocular Digitator
Posts: 9034
Joined: Wed Sep 12, 2007 12:42 pm
Location: Modesto, CA

Re: NASA Open API for APOD

Post by geckzilla » Fri Dec 13, 2019 1:17 am

just my two cents, but it's more important to actually have some classes and ids defined within the html than it is to have the css file itself.
Just call me "geck" because "zilla" is like a last name.

User avatar
RJN
Baffled Boffin
Posts: 1551
Joined: Sat Jul 24, 2004 1:58 pm
Location: Michigan Tech

Re: NASA Open API for APOD

Post by RJN » Sat Dec 14, 2019 7:21 pm

geckzilla wrote:
Fri Dec 13, 2019 1:17 am
just my two cents, but it's more important to actually have some classes and ids defined within the html than it is to have the css file itself.
Thank you. But why? Is it so that codes that ingest the HTML will better be able to figure out details of what information they are ingesting, like copyright information?

User avatar
geckzilla
Ocular Digitator
Posts: 9034
Joined: Wed Sep 12, 2007 12:42 pm
Location: Modesto, CA

Re: NASA Open API for APOD

Post by geckzilla » Sun Dec 15, 2019 6:31 am

RJN wrote:
Sat Dec 14, 2019 7:21 pm
geckzilla wrote:
Fri Dec 13, 2019 1:17 am
just my two cents, but it's more important to actually have some classes and ids defined within the html than it is to have the css file itself.
Thank you. But why? Is it so that codes that ingest the HTML will better be able to figure out details of what information they are ingesting, like copyright information?
Yes. If you've ever worked with databases or even a spreadsheet, in order to select a particular item in the data, it has to have *some* form of id. It's like that with HTML. The way to disambiguate every item within the page is to assign it an id. The same way CSS uses the id to select that item, so too can anyone who wants to parse the data select it based on the same id.
Just call me "geck" because "zilla" is like a last name.

PawelPleskaczynski
Asternaut
Posts: 9
Joined: Fri Nov 22, 2019 5:35 pm

Re: NASA Open API for APOD

Post by PawelPleskaczynski » Sun Dec 15, 2019 11:20 am

Sorry, I've been offline for a while, but I see that it's going in a good direction. I see that your intention is to make the website look the same (so straight from the 90s) but change the core code. In previous posts I suggested to rewrite everything from scratch, use a database and modern development tools. But this approach means that the website wouldn't work on older devices and browsers and existing web scraping tools, like official APOD API, or my own, would stop working. Your approach lets the web scrapers to work normally (I'll test if my APOD API would work on this test site) and it'll work on older browsers to some extent. Also please, really, please add classes or ids to elements on the website. Here's my suggestion; the APOD picture/video/something else would always have ID "apod-content", and other elements would always have ID describing what they are, eg. the title would have "title" ID and description field would have "copyright" ID. That way it'd be much, much easier to scrape this website. For now, to choose, for example, the copyright field, I have to do something like this: `copyright = body('center').eq(1).text();` which isn't elegant at all and it's prone to errors if website's layout changes. If elements had proper ID, this process would be as easy as something like this: `copyright = body('copyright').text();`.

videotizer
Ensign
Posts: 21
Joined: Thu Nov 21, 2019 10:24 pm

Re: NASA Open API for APOD

Post by videotizer » Sun Dec 15, 2019 12:02 pm

@geckzilla @PawelPleskaczynski

Did you take a look at my previous post? it covers all the points regarding the use of semantic elements, ids, and classes where applicable. I also created a revised home page which you can see here https://gist.github.com/videotizer/0493 ... e257678724.

PawelPleskaczynski
Asternaut
Posts: 9
Joined: Fri Nov 22, 2019 5:35 pm

Re: NASA Open API for APOD

Post by PawelPleskaczynski » Sun Dec 15, 2019 2:05 pm

@videotizer

Sorry, I didn't notice your post.. But yes, I fully agree with your points, and the website's code you provided looks good.

videotizer
Ensign
Posts: 21
Joined: Thu Nov 21, 2019 10:24 pm

Re: NASA Open API for APOD

Post by videotizer » Sun Dec 15, 2019 3:00 pm

@PawelPleskaczynski

No worries. Glad we agree.

User avatar
geckzilla
Ocular Digitator
Posts: 9034
Joined: Wed Sep 12, 2007 12:42 pm
Location: Modesto, CA

Re: NASA Open API for APOD

Post by geckzilla » Sun Dec 15, 2019 7:37 pm

videotizer wrote:
Sun Dec 15, 2019 12:02 pm
@geckzilla @PawelPleskaczynski

Did you take a look at my previous post? it covers all the points regarding the use of semantic elements, ids, and classes where applicable. I also created a revised home page which you can see here https://gist.github.com/videotizer/0493 ... e257678724.
yes, I saw it, and I wish you the same luck that I wish all people on this same endeavor. maybe you will be the lucky one.
Just call me "geck" because "zilla" is like a last name.

User avatar
RJN
Baffled Boffin
Posts: 1551
Joined: Sat Jul 24, 2004 1:58 pm
Location: Michigan Tech

Re: NASA Open API for APOD

Post by RJN » Mon Dec 16, 2019 4:31 pm

Another strange idea. Would it be possible to write a code (say, in Python) that ingests that current APOD HTML and outputs an HTML file that has all the useful semantic tags. Alternatively the output could be an XML file. Then Jerry and I could continue to write and edit (nearly) the same APOD HTML file but have automatically created a more ingestion-friendly file that APIs and the like can then utilize. The two APOD HTML files together would allow both ingesters -- one hard-coded to the old HTML, and the other newly more easily coded ingesting programs -- to work simultaneously. We at APOD central would alert mirror sites and world language sites that ingest the old HTML file that this file will soon go away, and so they should switch over to working with the new HTML file. However, they will have a grace period of a year or so where both APOD HTML files will exist simultaneously so that everything works for everyone.

Later, after the transition, we at APOD central can write/edit ONLY an XML file which will be ingested by a program (say, in Python) that generates the HTML file that is then picked up by browsers, mirror sites, smartphone apps, etc. Thoughts?

User avatar
Chris Peterson
Abominable Snowman
Posts: 14816
Joined: Wed Jan 31, 2007 11:13 pm
Location: Guffey, Colorado, USA

Re: NASA Open API for APOD

Post by Chris Peterson » Mon Dec 16, 2019 6:20 pm

RJN wrote:
Mon Dec 16, 2019 4:31 pm
Another strange idea. Would it be possible to write a code (say, in Python) that ingests that current APOD HTML and outputs an HTML file that has all the useful semantic tags. Alternatively the output could be an XML file. Then Jerry and I could continue to write and edit (nearly) the same APOD HTML file but have automatically created a more ingestion-friendly file that APIs and the like can then utilize. The two APOD HTML files together would allow both ingesters -- one hard-coded to the old HTML, and the other newly more easily coded ingesting programs -- to work simultaneously. We at APOD central would alert mirror sites and world language sites that ingest the old HTML file that this file will soon go away, and so they should switch over to working with the new HTML file. However, they will have a grace period of a year or so where both APOD HTML files will exist simultaneously so that everything works for everyone.

Later, after the transition, we at APOD central can write/edit ONLY an XML file which will be ingested by a program (say, in Python) that generates the HTML file that is then picked up by browsers, mirror sites, smartphone apps, etc. Thoughts?
If someone is going to sit down and do a bit of serious coding, I think the clean solution is to provide a tool where you or Jerry simply fill in a bunch of fields (could be a standalone program or one that runs on a web server) and outputs the XML file you're talking about. That XML file is the source document for an APOD. Then all you need is a simple renderer that reads that in and outputs HTML. Much simpler than going from HTML to XML and back to HTML again. It's likely to take longer to figure out what fields are necessary (although the API work has already done most of this) than to actually produce the code.
Chris

*****************************************
Chris L Peterson
Cloudbait Observatory
http://www.cloudbait.com

videotizer
Ensign
Posts: 21
Joined: Thu Nov 21, 2019 10:24 pm

Re: NASA Open API for APOD

Post by videotizer » Mon Dec 16, 2019 6:50 pm

Chris Peterson wrote:
Mon Dec 16, 2019 6:20 pm
I think the clean solution is to provide a tool where you or Jerry simply fill in a bunch of fields (could be a standalone program or one that runs on a web server)
If that's the case I would suggest using a proven Content Management System (CMS) such as WordPress. It has builtin features that cover most of what's needed, such as:
  • Posts management and editing with possibility for multiple authors
  • Taxonomy management for tagging and categorizing
  • Archive
  • User management
  • RSS feed
  • API
BTW, I noticed that some mirror/world language sites are already using WordPress, such as: If you're interested I'll be more than happy to setup a demo server.

User avatar
Chris Peterson
Abominable Snowman
Posts: 14816
Joined: Wed Jan 31, 2007 11:13 pm
Location: Guffey, Colorado, USA

Re: NASA Open API for APOD

Post by Chris Peterson » Mon Dec 16, 2019 7:10 pm

videotizer wrote:
Mon Dec 16, 2019 6:50 pm
Chris Peterson wrote:
Mon Dec 16, 2019 6:20 pm
I think the clean solution is to provide a tool where you or Jerry simply fill in a bunch of fields (could be a standalone program or one that runs on a web server)
If that's the case I would suggest using a proven Content Management System (CMS) such as WordPress. It has builtin features that cover most of what's needed, such as:
  • Posts management and editing with possibility for multiple authors
  • Taxonomy management for tagging and categorizing
  • Archive
  • User management
  • RSS feed
  • API
BTW, I noticed that some mirror/world language sites are already using WordPress, such as: If you're interested I'll be more than happy to setup a demo server.
I think that using WP is massive overkill here (and it's based on SQL databases, not XML. For something as simple as APOD, XML makes a lot more sense and it's a lot more portable. I run a number of WP based sites, and I'll say that there's a fair bit of overhead involved with keeping them up-to-date, and some updates break things. I'm sure that the editors don't want to deal with anything like that. It would border on trivial to write a form processor that extracted the fields to an XML file, and also to write a parser that converted the XML file to HTML. And these would be very stable, not dependent upon third party software like WP, and require essentially no maintenance.
Chris

*****************************************
Chris L Peterson
Cloudbait Observatory
http://www.cloudbait.com

videotizer
Ensign
Posts: 21
Joined: Thu Nov 21, 2019 10:24 pm

Re: NASA Open API for APOD

Post by videotizer » Mon Dec 16, 2019 7:54 pm

Chris Peterson wrote:
Mon Dec 16, 2019 7:10 pm
I think that using WP is massive overkill here (and it's based on SQL databases, not XML. For something as simple as APOD, XML makes a lot more sense and it's a lot more portable. I run a number of WP based sites, and I'll say that there's a fair bit of overhead involved with keeping them up-to-date, and some updates break things. I'm sure that the editors don't want to deal with anything like that. It would border on trivial to write a form processor that extracted the fields to an XML file, and also to write a parser that converted the XML file to HTML. And these would be very stable, not dependent upon third party software like WP, and require essentially no maintenance.
Hi Chris,

I agree that it has it's cons, just like everything else, but if you think long term and think about the issues related to maintaining the API among other things, then using a CMS would become more feasible then trying to come up with hybrid solutions that would not provide a proper centralized workflow - we don't need to reinvent the wheel. There's always something that needs to be maintained/updated/upgraded/etc..., even APOD in it's current HTML only form requires that. The question is what's the most efficient way to maintain and manage a particular website, and in the case of APOD, I believe it has grown large enough to require a proper CMS to achieve that task.

Perhaps if @RJN would explain the current workflow for posting the daily picture, it would become easier to compare different proposed solutions.

PawelPleskaczynski
Asternaut
Posts: 9
Joined: Fri Nov 22, 2019 5:35 pm

Re: NASA Open API for APOD

Post by PawelPleskaczynski » Sun Dec 29, 2019 4:00 pm

I don't really like where this is going. In my opinion, website shouldn't be modified, other than maybe upgrading to HTML5
(though I feel that's not very important to do, because the website would stop working on older browsers) and adding IDs to some sections, like title or description. If the current workflow of adding new APOD is convenient for admins, it makes no sense to change anything. Also, the website works pretty well with current APIs, including the official one (though official one is poorly maintained), changing the layout and using a CMS or other system would break them and they would need to be updated.

User avatar
RJN
Baffled Boffin
Posts: 1551
Joined: Sat Jul 24, 2004 1:58 pm
Location: Michigan Tech

Re: NASA Open API for APOD

Post by RJN » Mon Dec 30, 2019 3:04 am

PawelPleskaczynski wrote:
Sun Dec 29, 2019 4:00 pm
... website shouldn't be modified, other than maybe upgrading to HTML5 ... and adding IDs to some sections, like title or description.
Wow, that's pretty close to my view! In fact, a few weeks ago, I created a minimal-change, HTML5-compatible version of an APOD page. It can be found here: https://apod.nasa.gov/apod/fap/pagetest5.html . I have been thinking of adding a copyright ID tag, but as usual I worry about a domino effect to some of APOD's mirror sites.

When editing new APODs this past week, I remembered again why we edit the main HTML each day and not rely on some form or XML meta-version. It is because there are, many times, unusual changes, such as rollover images, videos, links to future lectures, and possible other things that are easily coded in right in the HTML but not just would not be easily incorporated into an XLM-type meta-page.

- RJN

User avatar
geckzilla
Ocular Digitator
Posts: 9034
Joined: Wed Sep 12, 2007 12:42 pm
Location: Modesto, CA

Re: NASA Open API for APOD

Post by geckzilla » Tue Dec 31, 2019 6:49 pm

Using templating or a CMS system doesn't necessarily mean you can't still do that, though. A widely distributed CMS requires constant updating, though, and that's what would get you in the end. That, or you'd get hacked eventually if you failed to update.

I'm just at a loss as to why even the most simple changes such as adding IDs and a css page can't just as easily be dealt with by the mirror sites. APOD literally breaks itself on a regular basis by human error, and everyone deals with it just fine.
Just call me "geck" because "zilla" is like a last name.

Gimmeslack12
Asternaut
Posts: 6
Joined: Mon Jan 31, 2011 4:58 am

Re: NASA Open API for APOD

Post by Gimmeslack12 » Tue Feb 11, 2020 4:08 am

Overall the API runs great and the options for it are generally good, I wish it had a search functionality to it, or the ability to query multiple, non-sequential dates. I also wish the concept_tags option was enabled to provide more categorical descriptions of the APOD's.

I have also discovered that you can get thumbnails of the APOD's by visiting the link: https://apod.nasa.gov/apod/calendar/S_<DATE>.jpg where the <DATE> is represented in the format: `031106` -> `YYMMDD`. This is the same date format as APOD's are represented as. Example:

Code: Select all

APOD Link
https://apod.nasa.gov/apod/ap031106.html

Thumbnail
https://apod.nasa.gov/apod/calendar/S_031106.jpg
For the past 2 years I've been using the APOD API to power my Chrome/Firefox Extension called "APOD By the Trav". It adds APOD as your new tab page with a number of (what I think) are cool features. I am very excited to share it wherever I can but don't want to be too spammy about it.

Chrome - https://chrome.google.com/webstore/deta ... pdcmlfjcdj
Firefox - https://addons.mozilla.org/en-US/firefo ... -the-trav/
MS Edge - https://microsoftedge.microsoft.com/add ... ngnlfipjdh

I am still pretty active in developing it and any thoughts or feature ideas anyone has I'm more than happy to hear about.

Gimmeslack12
Asternaut
Posts: 6
Joined: Mon Jan 31, 2011 4:58 am

Re: NASA Open API for APOD

Post by Gimmeslack12 » Tue Feb 11, 2020 4:44 am

Ok, I just realized the conversation that was happening in this thread after I posted my post above ^^^ about my extension (and that I wasn't really adding to the topic at hand). So I went and read through everything to understand what's going on.

All I'll say in regards to breaking older browsers is that if server side rendering from the API was supported then you could still output a minimally upgraded, HTML5 compliant, APOD site yet have the power of modern development tools. I didn't catch exactly how APOD is updated each day (maybe it's an old HTML template that's copied each day?) but that cut and paste approach could be deprecated with the use of a simple admin page form for adding the date/description/hdurl/url/etc. that uploads to the API database. I could even see the ability to line up a week or month of APOD's in advance and have that switch over at midnight EST (or whenever!).

Whatever the case, I'm a willing front end developer that is at your service.

User avatar
RJN
Baffled Boffin
Posts: 1551
Joined: Sat Jul 24, 2004 1:58 pm
Location: Michigan Tech

Re: NASA Open API for APOD

Post by RJN » Tue Feb 11, 2020 5:28 pm

Gimmeslack12 wrote:
Tue Feb 11, 2020 4:44 am
Whatever the case, I'm a willing front end developer that is at your service.
Thanks! Your offer is appreciated. Things may move on this front this summer.