Google App Engine Application Suddenly Stopped Working, returning 503

问题

I have an application running on Google App Engine, I deployed the last version of latest version of the application on Sep 6, 2018 but today the application stopped serving request and it is returning error 503 with the following

<HTML>
  <HEAD>
    <TITLE>Service Unavailable</TITLE>
  </HEAD>
 <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Service Unavailable</H1>
 <H2>Error 503</H2>
  </BODY>
 </HTML>

I thought it was because i have not migrated to the deprecated Cloud Endpoint v1.1 to V2.0

But after I made the changes necessary for migration, when I try to deploy it still returns 503 while running the Maven command to deploy or generate Docs.

I am aware that Google stated that they are experiencing issues with some App Engine regions yesterday but I am not sure if it has anything to do with what I am experiencing.

My App engine application is using Us-central region.

Update: Looks like cloud endpoint V1.1 has been turned off and can no longer serve traffic so following the migration guide, I have been able to migrate to version 2.0 but unfortunately I am getting an error with the api-discovery doc

This is the error

500

API discovery response missing required fields.

I am trying to get around this

here is my pom.xml file

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0        http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>

<groupId>group-id</groupId>
<artifactId>artifact-id</artifactId>

<properties>
    <app.id>project-id</app.id>
    <app.version>1</app.version>
    <appengine.version>1.9.64</appengine.version>
    <gcloud.plugin.version>2.0.9.121.v20160815</gcloud.plugin.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
    <jackson.version>2.9.0</jackson.version>
    <archiveClasses>true</archiveClasses>
</properties>

<prerequisites>
    <maven>3.3.9</maven>
</prerequisites>

<dependencies>
    <!-- Compile/runtime dependencies -->
    <dependency>
        <groupId>com.google.endpoints</groupId>
        <artifactId>endpoints-framework</artifactId>
        <version>2.1.0</version>
    </dependency>

    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>servlet-api</artifactId>
        <version>2.5</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>javax.inject</groupId>
        <artifactId>javax.inject</artifactId>
        <version>1</version>
    </dependency>

    <!--  Objectify library ready -->
    <dependency>
        <groupId>com.googlecode.objectify</groupId>
        <artifactId>objectify</artifactId>
        <version>6.0</version>
    </dependency>

    <!--  Apache Commons Library -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.0</version>
    </dependency>

    <!-- Add support for joda money library  -->
    <dependency>
        <groupId>org.joda</groupId>
        <artifactId>joda-money</artifactId>
        <version>0.10.0</version>
    </dependency>

    <!--Added support for joda time library -->
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.9.4</version>
    </dependency>
    <!-- end joda time api.-->

    <!-- Load scrypt library for password encryption and decryption -->
    <dependency>
        <groupId>com.lambdaworks</groupId>
        <artifactId>scrypt</artifactId>
        <version>1.4.0</version>
    </dependency>
    <!-- Objectify utils to help us with currency -->
    <dependency>
        <groupId>com.sappenin.objectify</groupId>
        <artifactId>objectify-utils</artifactId>
        <version>5.1.3</version>
    </dependency>
    <!-- google guava -->
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>19.0</version>
    </dependency>
    <!-- Apache poi for excel file processing -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>3.15</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>3.15</version>
    </dependency>


    <!-- jackson-dataformat-csv-->
    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-csv</artifactId>
        <version>2.7.0</version>
    </dependency>

    <!-- jackson data bind library for json -->
    <!-- Google cloud storage ends -->
    <!-- google cloud storage client -->
    <dependency>
        <groupId>com.google.appengine.tools</groupId>
        <artifactId>appengine-gcs-client</artifactId>
        <version>0.7</version>
        <exclusions>
            <exclusion>
                <groupId>com.google.api-client</groupId>
                <artifactId>google-api-client-appengine</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.google.http-client</groupId>
                <artifactId>google-http-client-jackson2</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.google.http-client</groupId>
                <artifactId>google-http-client</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-storage</artifactId>
        <version>0.4.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.api-client</groupId>
        <artifactId>google-api-client-appengine</artifactId>
        <version>1.23.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client-jackson2</artifactId>
        <version>1.23.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client</artifactId>
        <version>1.23.0</version>
    </dependency>
    <!-- Test Dependencies -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>2.0.2-beta</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.google.appengine</groupId>
        <artifactId>appengine-api-stubs</artifactId>
        <version>${appengine.version}</version>
        <scope>test</scope>
    </dependency>

    <!-- Mail Dependencies with MailGun -->
    <dependency>
        <groupId>com.sun.jersey</groupId>
        <artifactId>jersey-core</artifactId>
        <version>1.19.1</version>
    </dependency>
    <dependency>
        <groupId>com.sun.jersey</groupId>
        <artifactId>jersey-client</artifactId>
        <version>1.19.1</version>
    </dependency>
    <dependency>
        <groupId>com.sun.jersey.contribs</groupId>
        <artifactId>jersey-multipart</artifactId>
        <version>1.19.1</version>
    </dependency>
    <!-- End Mail Dependencies with MailGun -->
    <dependency>
        <groupId>org.freemarker</groupId>
        <artifactId>freemarker-gae</artifactId>
        <version>2.3.25-incubating</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>${jackson.version}</version>
    </dependency>

    <dependency>
        <!-- Note: core-annotations version x.y.0 is generally compatible with
             (identical to) version x.y.1, x.y.2, etc. -->
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>${jackson.version}</version>
    </dependency>

</dependencies>

<build>
    <!-- for hot reload of the web application-->
    <outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>versions-maven-plugin</artifactId>
            <version>2.1</version>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>display-dependency-updates</goal>
                        <goal>display-plugin-updates</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>        
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>appengine-maven-plugin</artifactId>
                <version>1.3.2</version>
                <configuration>
                    <stage.enableJarClasses>true</stage.enableJarClasses>
                    <cloudSdkPath>/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk</cloudSdkPath>
                </configuration>
            </plugin>
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>appengine-maven-plugin</artifactId>
            <version>1.3.2</version>
            <configuration>
                <!-- deploy configuration -->
            </configuration>

        </plugin>

        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>endpoints-framework-maven-plugin</artifactId>
            <version>1.0.2</version>
            <configuration>
                <!-- plugin configuration -->
                <hostname>project-id.appspot.com</hostname>

            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>discoveryDocs</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Here is my web.xml file

 <?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <!-- Endpoints web.xml -->
    <!-- [START web] -->
    <servlet>
        <display-name>Remote API Servlet</display-name>
        <servlet-name>RemoteApiServlet</servlet-name>
        <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet>
        <servlet-name>Grader</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.Grader</servlet-class>
    </servlet>
    <servlet>
        <servlet-name>SheetHandler</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.SheetHandler</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>RemoteApiServlet</servlet-name>
        <url-pattern>/remote_api</url-pattern>
    </servlet-mapping>
    <!-- Score Sheet Servlet -->
    <!-- Remote Api Servlet -->
    <servlet>
        <servlet-name>EndpointsServlet</servlet-name>
        <servlet-class>com.google.api.server.spi.EndpointsServlet</servlet-class>
        <init-param>
            <param-name>services</param-name>
            <param-value>com.softquest.apollo.services.UserServices</param-value>
        </init-param>
    </servlet>
    <!-- Payment Response Servlet -->
    <servlet>
        <servlet-name>PaymentResponseServlet</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.PaymentResponseServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>PaymentResponseServlet</servlet-name>
        <url-pattern>/PaymentResponse</url-pattern>
    </servlet-mapping>
    <!-- Payment Response Servlet-->
    <servlet-mapping>
        <servlet-name>EndpointsServlet</servlet-name>
        <url-pattern>/_ah/api/*</url-pattern>
    </servlet-mapping>
    <!-- Objectify filter registration -->
    <filter>
        <filter-name>ObjectifyFilter</filter-name>
        <filter-class>com.googlecode.objectify.ObjectifyFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>ObjectifyFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>
    <!-- Setup ServletContextListener to start up objectify entities on application -->
    <servlet>
        <servlet-name>BackEndStarter</servlet-name>
        <servlet-class>com.softquest.apollo.BackEndStarter</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>BackEndStarter</servlet-name>
        <url-pattern>/backend/starter</url-pattern>
    </servlet-mapping>
    <!-- Migration servlet -->
    <servlet>
        <servlet-name>MigrationServlet</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.MigrationServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>MigrationServlet</servlet-name>
        <url-pattern>/migration/*</url-pattern>
    </servlet-mapping>
    <!-- Migration Servlet-->
    <!-- Migration Handler Servlet -->
    <servlet>
        <servlet-name>MigrationHandler</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.MigrationHandler</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>MigrationHandler</servlet-name>
        <url-pattern>/migration-handler/*</url-pattern>
    </servlet-mapping>
    <!-- End of Migration handler Servlet-->
    <!-- Setup web.xml to start up context listener -->
    <servlet-mapping>
        <servlet-name>Grader</servlet-name>
        <url-pattern>/Grader</url-pattern>
    </servlet-mapping>
    <servlet-mapping>
        <servlet-name>SheetHandler</servlet-name>
        <url-pattern>/SheetHandler</url-pattern>
    </servlet-mapping>
    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
    </welcome-file-list>
</web-app>

I get this error when trying to call the api discovery doc, I have traced it to a null pointer error on line 72 of the

com.google.api.server.spi.EndpointsServlet file

if (!dispatcher.dispatch(method, path, context)) {
        response.setStatus(HttpServletResponse.SC_NOT_FOUND);
        response.getWriter().append("Not Found");
      }

I am guessing the error is released to some third party library probably still using any of the older cloud endpoint maven dependencies or plugins. So I would remove all the dependencies from my pom.xml and try to deploy.

UPDATE: I was able to fix the API discovery error, it was caused by a reflection error that had to do with the Google Guava Library, while the migration Doc had stated that Version 19 would work, it didn't, upgrading to version 20 and above fixed that for me, now I am getting error 404 on all the endpoints. I would write an answer when I am able to fix this.

回答1:

Im getting the same problem on an endpoint that hasn't been changed in months, nothing is appearing in the logs either. I assume its probably an app engine problem.

EDIT

I fixed our problem, though you mention the you've attempted to update to the new Cloud Endpoint v2.0 and you are still having the problem it turns out for our project following the migration steps from v1.1 to v2 fixed the problem https://cloud.google.com/endpoints/docs/frameworks/python/migrating.

The docs say v1.1 was to be turned off August 2nd, I guess they have just got round to it

回答2:

I have successfully migrated to Endpoints V2 after series of hit and trial method.

App Engine project developed on Android Studio has following structure-

project

app
backend
build.gradle

I have to change build.gradle in all 3 places. Where project is root project, app is android app, backend is App Engine app. All of these have thier own build.gradlew.

Let's change one by one.

/app/build.gradle (Make changes in the android client)

dependencies {
compile fileTree(dir: 'libs', include: ['*.jar'])
compile 'com.android.support:appcompat-v7:26.+'

// Remove this line
//compile project(path: ':micirclews', configuration: 'android-endpoints')

}

Apply the endpoints plugin after the android plugin

apply plugin: 'com.google.cloud.tools.endpoints-framework-client'

/backend/build.gradle (Make changes in the backend/appengine module)

buildscript {
repositories {
    mavenCentral()
    jcenter()
}
dependencies {
    //Remove this
    //classpath 'com.google.appengine:gradle-appengine-plugin:1.9.59'

    //Add these two lines
    classpath 'com.google.cloud.tools:endpoints-framework-gradle-plugin:1.0.2'
    classpath 'com.google.cloud.tools:appengine-gradle-plugin:1.3.3'
}

}

Remove this- apply plugin: 'appengine'

Add these two-

apply plugin: 'com.google.cloud.tools.appengine'
apply plugin: 'com.google.cloud.tools.endpoints-framework-server'


dependencies {
//Remove these 3 lines-
//appengineSdk 'com.google.appengine:appengine-java-sdk:1.9.59'
//compile 'com.google.appengine:appengine-endpoints:1.9.59'
//compile 'com.google.appengine:appengine-endpoints-deps:1.9.59'

//Add these 2 lines-
compile 'com.google.endpoints:endpoints-framework:2.0.9'
compile 'javax.inject:javax.inject:1'
.....

}

// delete this whole block, it's configuration for the older plugin
// appengine {
//   downloadSdk = true
//   appcfg {
//     oauth2 = true
//   }
//   endpoints {
//     getClientLibsOnBuild = true
//     getDiscoveryDocsOnBuild = true
//   }
// }

/build.gradle (Make changes in the root's build.gradle)

buildscript {
repositories {
    jcenter()
}

dependencies {
    classpath 'com.android.tools.build:gradle:2.3.3'

    //Add these 2 lines
    classpath 'com.google.cloud.tools:endpoints-framework-gradle-plugin:1.0.2'
    classpath 'com.google.cloud.tools:appengine-gradle-plugin:1.3.3'
}

}

After making all the changes carefully, Clean the project, Rebuild it and run it. Finally deploy it when everything work perfect.

You may refer the whole process here - https://github.com/GoogleCloudPlatform/endpoints-framework-gradle-plugin/blob/master/ANDROID_README.md

回答3:

As saiyr stated in the comments, Google Cloud Endpoints v1.1 has been completely deprecated and has stopped serving traffic and that is the reason for the errors I was having. I solved this problem by migrating to cloud endpoints v2, the process is as easy as following the guide in the migration docs but here are some issues I had with Migration and how I fixed it.

Google guava library has to be updated to a version greater than version 19, using version 19 or lower could result in the application throwing reflection errors when handling requests.
If you have multiple service classes they have to be added to the web.xml file as service parameters while declaring the com.google.api.server.spi.EndpointsServlet Servlet. for some reasons listing only one service class worked for me in cloud endpoint 1.0 Api docs and routes were generated for all the service classes I had without declaring everyone of them explicitly under the com.google.api.server.spi.SystemServiceServlet Servlet, I think it has something to do with Objectify filters though. See the Docs for more info
Since the Google App Engine Java 7 runtime is deprecated too, I decided to migrate to Java 8 but unfortunately the library I used to abstract the process of connecting to datastore (Objectify) appears not to work with the Java 8 runtime, I would look into this later, but reverting back to Java 7 runtime worked for me and the application is back up and running.

Update: The issue I faced migrating to Java 8 runtime had nothing to do with Objectify, the library I used for encrypting passwords (BCrypt) was the issue, it just did not work on the Google App Engine Java 8 runtime, I had to switch to the Encryption library in spring security.

来源：https://stackoverflow.com/questions/52312107/google-app-engine-application-suddenly-stopped-working-returning-503

标签

google-app-engine

google-cloud-platform

google-cloud-endpoints

google-cloud-endpoints-v2