tag:status.smarty.com,2005:/historySmarty Status - Incident History2024-03-19T08:48:51ZSmartytag:status.smarty.com,2005:Incident/201973442024-03-08T22:49:43Z2024-03-08T22:49:43ZInternational Address Autocomplete API: Configuration Fix Resolves 402 Response Issue<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>22:49</var> UTC</small><br><strong>Resolved</strong> - A configuration change to an older version of International Address Autocomplete API caused a few users to receive inaccurate 402 responses. The configuration was identified and rectified. Users of the older version of this API should no longer be receiving the erroneous response. Users of the new version of the API were unaffected.</p>tag:status.smarty.com,2005:Incident/201386682024-03-01T23:11:15Z2024-03-01T23:11:15ZOlder applications affected briefly by TLS modifications<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Resolved</strong> - Some modifications were made to TLS certification configurations on one of our load-balancers. Part of the modification adjusted the order in which certificates were presented to clients. Older applications that are unable to properly use Server Name Indication may have been affected adversely by this change. In order to continue supporting older clients, the certificate order was readjusted to maximize compatibility.<br /><br />We suggest updating any applications that call Smarty APIs to be compatible with modern TLS certificate usage.<br /><br />We also suggest ensuring that the hostnames being used in requests to our APIs are current. Hostnames can be found on the documentation pages for the products that you are using. (https://www.smarty.com/docs/cloud)</p>tag:status.smarty.com,2005:Incident/201115282024-02-29T17:30:00Z2024-02-29T18:35:11ZOne datacenter returned 503 responses briefly<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Resolved</strong> - During a regularly scheduled deployment of an update to our US Address Verification API and US Address Autocomplete API in one of our datacenters, a misconfiguration caused the services to not respond properly. The problem was quickly observed and remedied.</p>tag:status.smarty.com,2005:Incident/198783142024-01-31T18:56:39Z2024-01-31T19:01:44ZJSONP support removed<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>18:56</var> UTC</small><br><strong>Resolved</strong> - As previously announced, JSONP support in US Street Address API is being removed today.</p>tag:status.smarty.com,2005:Incident/198803402024-01-22T19:00:00Z2024-02-01T00:22:28ZIntermittent 503 responses from US Street Address API<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Resolved</strong> - On 2024 January 22 US Street Address API experienced some unexpected resource constraints that briefly caused a few intermittent 503 errors for a handful of customers. These restraints have been resolved.</p>tag:status.smarty.com,2005:Incident/191304042023-11-15T19:57:31Z2023-11-15T19:57:31ZBREAKING CHANGE: International [Address] Autocomplete API<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Resolved</strong> - A new endpoint with new capabilities has been released. As a result of this change, the older (existing) endpoint is being deprecated and will be removed from service.<br /><br />Please see our documentation regarding this mandatory and breaking changes:<br />- https://get.smarty.com/intl-autocomplete-migrate<br />- https://www.smarty.com/docs/cloud/international-address-autocomplete-api</p>tag:status.smarty.com,2005:Incident/191198542023-11-14T17:22:47Z2023-11-14T17:22:47ZJSONP is officially deprecated<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>17:22</var> UTC</small><br><strong>Resolved</strong> - JSONP is officially deprecated. Support will be removed for JSONP starting on 2024 January 31.</p>tag:status.smarty.com,2005:Incident/190390882023-11-06T21:39:33Z2023-11-06T21:39:33ZUS Street API - Intermittent HTTP 400 Responses<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>21:39</var> UTC</small><br><strong>Resolved</strong> - During the last three days we have been observing a significant increase in CPU utilization of various application servers. As a result of this CPU utilization, we made some adjustments to the software running our proxy API servers to better watch for traffic causing the CPU spikes.<br /><br />As part of this remediation effort, at 2023-11-05 00:54 UTC, a new version of our proxy API with this new behavior was deployed to a single cluster. After the “canary” release appeared to be functioning correctly according to available metrics, we rolled the change out to a second cluster roughly 15 minutes later. Finally, roughly 21 hours later, at 2023-11-05 21:45 UTC nearly, we rolled out the change to the remaining third cluster.<br /><br />At 2023-11-06 01:40 UTC, we received notice from approximately four different customers of some intermittent issues wherein an HTTP request to our US Street API (https://us-street.api.smarty.com) would return an HTTP 400 response status. Within five minutes after that notice, at 2023-11-06 01:45 UTC, we rolled back the above changes to a working version of the proxy API across our entire fleet.<br /><br />Since that time, the underlying bug within the proxy API introduced and originally deployed at 2023-11-05 00:54 UTC was resolved and a new version of the software that is unaffected by this issue was deployed to a single cluster today at 2023-11-06 16:50 UTC.<br /><br />Additional tests and monitoring have been put in place to proactively watch for similar kinds of issues in the future.</p>tag:status.smarty.com,2005:Incident/178371382023-07-12T14:00:00Z2023-07-12T19:16:11ZRoot Cause Analysis of IP Address Failure<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - One of our IP addresses became unresponsive. Investigation showed that the IP was configured properly, however the connection of the IP, to the load balancer to which it was assigned, was incomplete. This situation started when one of our nodes went offline and the IP was automatically assigned to a different node (a common and normal occurrence). When the node that went offline came back online, the IP was reassigned to the original node (also normal). While the assignment was correctly executed, the IP provider did not properly complete the routing of traffic (abnormal and unusual). As we do not control the IP routing at the provider level, we are implementing additional monitoring checks that will allow us to detect and mitigate this unusual situation in the future.</p>tag:status.smarty.com,2005:Incident/171256912023-05-02T22:00:00Z2023-05-02T22:48:15Z404 Not Found<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Resolved</strong> - Some customers experienced an occasional 404 "Not Found" error with the new update that we pushed yesterday afternoon to one server set and rolled back this morning. This was due to some requests formatted with a slash after the route "street-address/". e.g. "https://us-street.api.smartystreets.com/street-address/?input_id" vs. "https://us-street.api.smartystreets.com/street-address?input_id". This was not recognized correctly in the new update and will be corrected.</p>tag:status.smarty.com,2005:Incident/169146072023-04-18T20:30:00Z2023-04-19T20:45:14ZUS Autocomplete Pro API: addresses not grouping<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>20:30</var> UTC</small><br><strong>Resolved</strong> - Between Apr 18, 2023 2:30pm and Apr 19, 2023 10:30am the us-autocomplete-api was not grouping sub-units (apartments, suites and so forth). This has been corrected.</p>tag:status.smarty.com,2005:Incident/167412472023-04-03T18:00:00Z2023-04-04T00:39:34ZInternational Street API reported inconsistencies<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Resolved</strong> - On Friday we deployed a minor system update. Beginning on Monday morning a few customers reported inconsistencies in the International Street Address API output. We rolled back the update and the inconsistencies have stopped. We are still investigating the root cause.</p>tag:status.smarty.com,2005:Incident/165963752023-03-21T15:00:00Z2023-03-21T20:15:02ZDNS Update<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Resolved</strong> - A new IP address was introduced into our infrastructure for one of our facilities and a previous IP address was phased out of use. Part of this process involved updating DNS records. All appropriate records were updated with the exception of several legacy hostnames, which were overlooked. This resulted in a few users seeing intermittent request failures when requests resolved to the previous IP address. This only affected one of our facilities. All other facilities maintained full accessibility. The aforementioned hostnames have now been updated.</p>tag:status.smarty.com,2005:Incident/164232602023-03-10T17:40:33Z2023-03-10T17:40:33ZUpcoming TLS Certificate Rotation<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>17:40</var> UTC</small><br><strong>Resolved</strong> - On April 17, 2023 the current certificate for *.api.smartystreets.com will expire. This certificate was issued by Sectigo (previously known as Comodo) and is chained to the root certificate "USERTrust RSA Certification Authority".<br /><br />On March 21, 2023 at 12pm Eastern Time (approximately 2 weeks hence), we will begin the process of rotating out this soon-to-expire certificate with a newer certificate issued by a service chained to one of our trusted certificate authorities, as listed in our documentation [1]. Specifically, we intend to begin using certificates issued by various ACME-compliant certificate providers. In this case we will begin using certificates with the industry standard 90-day expiration from a company called ZeroSSL. These new certificates will continue to be chained to the same root CA as our soon-to-expire certificate issued by Sectigo.<br /><br />Following this, on April 18, 2023 we will begin to introduce certificates issued by Let’s Encrypt (chained to “ISRG Root X1”) and Google (chained to various "GTS Root" certificates).<br /><br />FAQ:<br /><br />Q: Will I need to take any action?<br />A: If you are "pinning" our certificate (meaning hard coding the certificate into your application, runtime (e.g. Java), or operating system "trust store"), yes. Please be aware that our certificates will now rotate on a much more frequent basis. In the past, the certificates were rotated yearly. Now they will be rotated at least every 60 days and often as frequently as every 30 days.<br /><br />Further, if you are manually curating which root certificate authorities you allow into your trust store, you will need to ensure that those listed in our documentation [1] are added to your trust store.<br /><br />Q: Why rotate the certificates so frequently? Isn’t that insecure?<br />A: It is considered an industry best practice to rotate certificates as frequently as reasonably possible. Doing so actually increases security and drastically decreases the likelihood of private key exposure. All automated (ACME-based) certificate providers typically have certificates expire after 90 days. In fact, in 2020 Apple began to enforce a 397 day limit on all certificates meaning that those with an expiration longer than 1 year and 1 month would be considered invalid. In other words, certificates with longer expiration dates are now considered to be less secure than those with shorter expiration dates.<br /><br />Q: What if I have more questions.<br />A: Please reach out to our support team for further clarification if you have any questions on the matter.<br /><br />References:<br />[1] https://www.smarty.com/docs/cloud/requirements#trusted-authorities</p>tag:status.smarty.com,2005:Incident/161406122023-02-14T19:00:00Z2023-02-14T22:32:37ZUS Rooftop Geocoding API: mid-cycle geocode update<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Resolved</strong> - Between 2023 January 16 and 2023 February 07, the US Rooftop Geocoding API returned inaccurate geocodes for a small set of addresses that were designated with a rooftop, parcel, or street-level geocode precision.<br /><br />The following addresses were affected:<br />- non-postal<br />- USPS R7<br />- non-postal addresses that became postal addresses before 2023 January 16<br /><br />The issue occurred due to incorrect temporary keys used to associate non-postal addresses to geocodes during the data build. The problem has been resolved and measures have been implemented to prevent this from happening in the future.<br /><br />Based on analysis, the inaccurate geocodes would likely have affected about 1.4% of the addresses processed by our clients.</p>tag:status.smarty.com,2005:Incident/158957952023-01-20T20:00:00Z2023-01-20T23:06:26ZCertificate Rotation<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Resolved</strong> - During a routine rotation of TLS certificates on our load balancers, a single load balancer in a single facility did not update properly. That instance was manually rotated to pick up the change.</p>tag:status.smarty.com,2005:Incident/126167412022-10-25T20:24:44Z2022-10-25T21:40:46ZInternational Street API intermittent connectivity<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>20:24</var> UTC</small><br><strong>Resolved</strong> - During the night the International Street API service had intermittent connectivity problems. We believe we have identified the problem and implemented a fix.</p>tag:status.smarty.com,2005:Incident/115001922022-10-05T22:15:11Z2022-10-05T22:15:11ZIntermittent errors on legacy US Autocomplete API service<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>22:15</var> UTC</small><br><strong>Resolved</strong> - The deprecated US Autocomplete API service underwent the process of termination over the course of yesterday and today. During the process, a few intermittent 503 responses were reported from customer who were still using the old address for calling the API. The address has been updated to send traffic to the US Autocomplete Pro API service. The update process for one of the loadbalancers handling that transition was out of sync. The problem was identified and remedied.</p>tag:status.smarty.com,2005:Incident/100118962022-05-17T15:31:58Z2022-05-17T15:31:58ZInternational Street API service<p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Resolved</strong> - The problem was found and corrected. All datacenters are fully operational again.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:27</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>14:23</var> UTC</small><br><strong>Investigating</strong> - In two of our datacenters, our International Street API service is in a degraded state. We are currently investigating.</p>tag:status.smarty.com,2005:Incident/72475682021-06-14T21:00:50Z2022-01-14T19:57:34ZDatacenter IP Address Changed<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - As of 14:25 (Mountain Time) today we migrated one of our facilities (US West Coast) to a new IP address. This update should not affect any of our customers that are following our recommended Technical Requirements (https://www.smartystreets.com/docs/cloud/requirements).<br /><br />Any customers who are hard-coding our IP addresses, without going through our Forward Proxy API (https://www.smartystreets.com/docs/cloud/forward-proxy-api), could be affected adversely.</p>tag:status.smarty.com,2005:Incident/66085302021-03-25T20:02:14Z2022-01-14T19:57:34ZWebsite outage<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>20:02</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Update</strong> - All systems are operational. We are still monitoring.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>15:53</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>15:51</var> UTC</small><br><strong>Monitoring</strong> - The problem is related to third-party DNS issues. We have implemented a temporary change which bypasses the failing DNS.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>15:12</var> UTC</small><br><strong>Identified</strong> - Our website is currently down. We are working to resolve the problem as quickly as possible. Our API services are unaffected..</p>tag:status.smarty.com,2005:Incident/65538782021-03-17T16:59:37Z2022-01-14T19:57:34ZInternational Street API latency post mortem<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Resolved</strong> - During Thursday 11 March 2020 and Saturday 13 March 2020 we noticed higher than normal latencies in our International Street API endpoint in two of our facilities.<br /><br />The problem was found to be coming where the default configuration in these specific facilities used slower, magnetic hard disk drives (HDD) instead of faster, solid state drives (SSD). This combined with a large influx of new requests from customers to cause the latency. The servers were quickly changed to all use solid state drives and the latency returned to normal.<br /><br />We have updated the configuration to explicitly require high-performance solid-state drives for the International Street API endpoint.</p>tag:status.smarty.com,2005:Incident/65170582021-03-12T19:53:11Z2022-01-14T19:57:35ZMinor bug prevented requests from legacy domains<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>19:53</var> UTC</small><br><strong>Resolved</strong> - During the course of a slow rollout of a newly rewritten authentication component, the software was deployed with an unintended bug which prevented the usual proxying of requests to the intended back-end service. The cause of the failure is due to an oversight on our part to register an old/legacy domain names still in use by affected customers (autocomplete-api.smartystreets.com as well as extract-beta.api.smartystreets.com). We have since re-added support for these domains and will continue with our slow rollout. Customers should update to the canonical hostnames for respective services (us-autocomplete.api.smartystreets.com and us-extract.api.smartystreets.com).</p>tag:status.smarty.com,2005:Incident/64908432021-03-10T20:12:19Z2022-01-14T19:57:35ZTLS Certificate Update<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>20:12</var> UTC</small><br><strong>Resolved</strong> - The update is complete.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>19:40</var> UTC</small><br><strong>Monitoring</strong> - The TLS certificate for our API services has been renewed. The updated certificate will be rolled out to all of our services over the next 24 hours.<br /><br />The updated certificate has a new root path. For details on our Trusted Certificate Authorities, review our requirements page:<br /><br />https://smartystreets.com/docs/cloud/requirements#trusted-authorities</p>tag:status.smarty.com,2005:Incident/63897072021-02-24T05:30:00Z2022-01-14T19:57:35ZDatacenter with above average latency removed<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:30</var> UTC</small><br><strong>Resolved</strong> - One of the datacenters that we use was exhibiting an above average latency. That datacenter has been removed from service until the latency issue is resolved.</p>