API Webinar "Q&A"¶

What is the API performance details?
Dennis (API Expert):

• It allows 10-80 concurrent requests
• Returns time less than 5 mins as per requirement
• Requests are processed sequentially

What is the session timeout (bearer token)?

• Dennis (API Expert): The token expires after 7 days

What are the equivalent endpoints of SFTP services?

• Dennis (API Expert): There is only one endpoint (/third-party) - multiple search options (refer to Redmine for document - https://redmine.uscdcb.com/attachments/18051)

How to know when a request has been processed?

• Dennis (API Expert): It is synchronous endpoint

Are CDCB using ADE standards or is this custom to CDCB?

• Dennis (API Expert): It is custom to CDCB requirements

While submitting the request for the third party, do we need to submit with pagination?

• Dennis (API Expert): No, so far I have not seen any pagination parameters for the third party API.

Are requests processed sequentially across all users?

• Dennis (API Expert): From the answer I got from the development team for this API, the subsequence of requests will apply for each user.

Is there any plan to decommission SFTP in place of the third-party API?

• Ezequiel (CDCB): Yes, the plans are, once times are matured the SFTP queries will be completely decommissioned and we will only work through the third-party API. The idea is to give 6 months time to people to transition out of the SFTP. So we will be communicating with those users that currently have the SFTP queries in place to set the time and the clock and make sure that everything is working before we stop that service.

Does the user ID you supply determine what information you can retrieve?

• Ezequiel (CDCB): This is related to security, so “Yes”. The type of user has certain access to certain queries.

What about accessing the API via unix terminal instead of using Postman software. Is it possible?

• Dennis (API Expert): You can use cURL to test the API, but for this API, I think it’s possible but in my capability, I cannot do it. However, if you have the capability to do it you can.

Can you explain in simple terms what the benefit of API is over SFTP?

• Ezequiel (CDCB): The idea is to decommission the SFTP queries that are in place for some users and not the SFTP itself. So files and results currently will still be released through the SFTP, as well as, evaluation service files. The only thing we plan to change are the SFTP queries that were an exception and created for some users while we didn’t have the API. The idea is that currently in the SFTP (referring to SFTP queries) a user puts a file (fix_fmt file) in a folder and an automation process runs on the backend to pick up those files which does a specific service to create some specific output. From there, it returns that back with a result. That is unnecessary now because with this new API service, it does all of that; and for those users that don’t have that API capability, there is a Data Exchange where they can achieve the same using a graphical UI. So essentially, we are decommissioning that because it’s a redundancy that is just creating overhead. In addition to that, for us it’s much easier to control one single situation, which is the API because both Data Exchange and API use the same engine instead of having different sources of variation here and there.

Will there be any routine/recurring outages, such as during evaluation runs? Is there a way to know if the service is available/unavailable programmatically?

• Ezequiel (CDCB): No, the idea and the design of Webconnect was purposely made to be more complicated on our end and make it more easier on your end. You will still be able to submit records and be able to see the database being functional because the web database is a separate database from our production now and we are syncing the two. So, that should reduce the outages linked to backups, for example, and things like that, so you will still be able to function. The only thing that will continue to happen is of course during freezes, you will not be able to. Those files will not be able to be processed until the database and the production database is unfrozen. During those days you will get a notification on the banner. We typically use that banner to communicate outages or queries not working, or to provide any communication that is important for all users to know. So the idea is pretty much what we did when we transitioned out from the old website that was local in a server that whenever we had an issue it was completely black and now the website is always open. This is the same for WebConnect and WebConnect services, unless a server breaks or needs to be rebooted it will always be open. It’s just the matter of processing those files that will continue to be blocked for three days a month for the freeze of the monthlies.

How often is the Webconnect database synchronized with the backed CDCB database?

• Ezequiel (CDCB): 24/7, This is an ongoing sync. It’s an IBM software that links the two databases. So, every time there is a change on the production database that’s immediately submitted to the Webconnect database. Of course, there is some data that is not synced to Webconnect. For example, genotypes are not put there because there is nothing that has to do with genotypes in Webconnect. So, that data is pretty much, I’m not saying that its immediate, but it's very very fast so you should notice a difference between that.

What is the supported RPS (Request Per Second)? Is the 10-80 per user or all users

• Dennis (API Expert): 10 - 80 requests per second

If we like to obtain more technical support on implementing the CDCB API’s who do we contact? What about coding for the API?

• Ezequiel (CDCB): You should put a ticket into Redmine. If this is something linked to data, the queries, or how the queries interact with your system, that would be CDCB. If it is something that we do not know how to answer that is much more technical, then we will create a request for DataGene and we will get back to you. So my answer to this question would definitely be to put a ticket into Redmine in order to be able to provide you with answers. In terms of coding, if it is something like the queries not working or you have questions about how to implement certain requests, that would be CDCB. If it is coding about the API, I would say between Google and your experienced developer.

Will allied industries need to transition to API’s in the future to maintain effective data exchange with CDCB?

• Ezequiel (CDCB): Definitely CDCB collaborators will benefit from interacting with the API. The reason for that is they can have a one-to-one interaction with our system directly. In terms of allied industries (i.e. NAAB), that would probably be a good idea. But please be aware that currently for the usage that you do, I see that NAAB has peaks of usage close to the closing windows of the automation and when that happens, definitely the interaction with the API or UI (Data Exchange) can be extremely beneficial. The reason being, is that you don’t have to go to a site, you can just send a file through the computer and get a response. So that is probably a much more efficient process for NAAB. Definitely the API interaction will grow in the future so this is just the first step and we will even have time for the industry to adapt to that before changing it. So this is definitely one way that we will be exploring and including much more in the future. So, my answer to this question is that it would be a good idea to start getting acquainted with the API and now that we have the resources to provide support, it’s also a better idea to get that ball rolling.

Is it your expectation that the nominators who currently use the web queries extensively will transition to using the API? Or should we be sharing this information with our IT teams for promote collaboration between them and CDCB?

• Ezequiel (CDCB): That is definitely our hope. The idea is that the API provides a much more large capability in terms of interaction with our system which gives you much more flexibility than we currently have, but of course requires on your end some more organization and some more programming on the backend in order to be able to interact with our system. The graphical queries that you are used to interacting with is limited to 50 animals, so if you are in need to change a specific animal, it’s much easier to go and do that change for those 20, 30, 40 animals. However, if you are checking a submission of a thousand animals, definitely that would make your life easier to just use a program that can just point out the animals that have issues and someone focusing on them instead of going and clicking on each one of them. So yes, the expectation is that definitely we will gradually transition towards the API. My suggestion would be to encourage that interaction between your IT teams and CDCB to get that collaboration started and yourself acquainted with the system, how it works, and if everything is in place because remember this is a new system that was built from scratch so there might be still things that need to be tweaked. That’s why we are in this very long transition and have been extending the transition period of the legacy system. We want to make sure that once we turn that off, you will be able to do everything that you used to do in the previous system in the new system and actually expand the possibilities in this new system compared to the previous one, in some cases.

Can you remind us of the general limit on the number of animals per query/API call?

• Ezequiel (CDCB): The limitation for the queries is 50. A single API request is limited in the number of requests to 2,000. The hourly/daily API request limitation is 10,000. So that’s a big indication of what we want you to use in the future. So that’s a big difference.

Are simple get request like getIDs going to be synchronous?

• Ezequiel (CDCB): All queries are asynchronous except for third-party.

Then does/would the 6 month transition take place, before the queries are decommissioned?

• Ezequiel (CDCB): The transition period for the SFTP queries and API is six months. It started when we launched (May 23rd), so we should be around half-way through. Two months before the deadline we will start communicating, one month before the deadline we will continue to communicate, and then closer to the deadline we will keep sending you reminders about that. The legacy system is expected to be decommissioned the moment we are running for a few weeks without major incidents like the ones we’ve had; and by incidents I mean services that need to be down until we fix a bug or an issue or we correct something. For example, we are about to resume the Fmt1 processing within a few days, so that should get to the second stage in which we get issues that should be less structural and that will essentially start the clock for us to see when to decommission the old queries.

Does “third_party” surface the existing requests like getids, or is limited to a subset of available queries?

• DataGene: There are 11 groups and each group has several search options, each option has several queries. For example:  the animal ID (12 bytes) in the ID/Pedigree group has 2 queries available which are Get Animal ID by 12 character ID number and Get Interbull ID from registration number. Please refer to the “List of Features and Queries” document in Redmine, which can be found here (https://redmine.uscdcb.com/attachments/18051).

In terms of larger scale data exchange for queries, we can utilize Postman or the Webconnect UI Data Exchange for files with more than 50 animals?

• Ezequiel (CDCB): Yes, the UI Data Exchange is essentially the same thing as the API, it’s just facilitated to you for your interaction and has a limitation on the return of results, which is 2000 records per request.

• Are submissions (genotype and format 1) uploads going to use API also?*
• Ezequiel (CDCB): Yes, that is going to be one of those evolutions that I was talking about in a few years from now but not immediately. We are not allowing currently any write to our infrastructure. This is one way, which is going from us to you. Format 1s follow that same logic. Hopefully in the future we will not have format 1s so you will be able to provide pedigree updates without fix formats, that’s in progress and there is the idea of developing something like this. But in the meantime you can change the IDs, so if you have a handful of animals that need to be changed on the pedigree you can use the queries in Webconnect to change a sire or a dam, multi-birth code (MBC), or date of birth (DOB). So the pedigrees can be changed directly from the query currently if you have the right permissions to edit that animal. You will be able to do that directly without sending a fmt1. Of course, if you are Holstein, for example, and you need to send 200,000 animals then that’s probably not the way to do it but if you need to correct a handful of animals or a specific animal then you can do that directly from Webconnect interactively.

So do we need to use hybrid —> rest for read and SFTP for writes?

• Ezequiel (CDCB): Yes, that is correct. So for submission of genotypes you will still need to be placing files in the SFTP just like you have been doing so far. But if you are referring to SFTP queries, then my suggestion is to use the rest for read and discontinue the use of SFTP as soon as possible because that will be discontinued.

But for updates to pedigree, we can still use fmt1 via SFTP until another couple of years?

• Ezequiel (CDCB): Correct, yes, the submission of the “In folder” is exactly the same as it is right now, there is no change on that. The only thing that will change is the queries, but the submission of files both fmt1s and for the DRPCs fmt4s, fmts 5 and 6 for the labs, and the genotypes. So everything that goes into the “In folder” will not change at all and the report of our files in the “Out folders” will not change at all. The only thing that will change is any other folder, well the quick turnaround for those that are using that service will not change but all the other folders like getgtype for some of you and things like that is going to be discontinued. This is the same for the “Check folder” as well.