问题
H2O process crashed when doing a Grid Search with XGBoost:
terminate called after throwing an instance of 'thrust::system::system_error' what(): /tmp/xgboost/plugin/updater_gpu/src/device_helpers.cuh(387): an illegal memory access was encountered
After giving the INFO message below:
08-17 06:44:46.672 10.0.1.89:54321 14426 FJ-1-3 INFO: Checking convergence with logloss metric: 0.04519170911104479 --> 0.02811784326194906 (still improving) . 08-17 06:44:46.672 10.0.1.89:54321 14426 FJ-1-3 INFO: For grid: final_grid built: 90 models.
The Java exception dumps:
08-17 06:44:46.742 10.0.1.89:54321 14426 #12317-18 INFO: GET /99/Grids/final_grid, parms: {} 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: java.lang.IllegalArgumentException: Field not found: 'col_sample_rate_change_per_level/_col_sample_rate _change_per_level' on object hex.tree.xgboost.XGBoostModel$XGBoostParameters@49356589 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.util.PojoUtils.getFieldValue(PojoUtils.java:562) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at hex.grid.Grid.createSummaryTable(Grid.java:370) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at hex.schemas.GridSchemaV99.fillFromImpl(GridSchemaV99.java:158) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.api.GridsHandler.fetch(GridsHandler.java:41) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at java.lang.reflect.Method.invoke(Method.java:498) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.api.Handler.handle(Handler.java:63) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.api.RequestServer.serve(RequestServer.java:448) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.api.RequestServer.doGeneric(RequestServer.java:297) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.api.RequestServer.doGet(RequestServer.java:221) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:183) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:183) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.Server.handle(Server.java:370) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:49 4) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53 ) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:9 71) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpCo nnection.java:1033) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: at java.lang.Thread.run(Thread.java:748) 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: Caught exception: 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: ERROR MESSAGE: 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: 08-17 06:44:46.747 10.0.1.89:54321 14426 #12317-18 ERRR: Field not found: 'col_sample_rate_change_per_level/_col_sample_rate_change_per_level' on object hex.tre e.xgboost.XGBoostModel$XGBoostParameters@49356589
回答1:
col_sample_rate_per_level is not supported for xgboost, only GBM and random forest:
http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/col_sample_rate_change_per_level.html
Here is the list of what you can use with an xgboost grid: http://docs.h2o.ai/h2o/latest-stable/h2o-docs/grid-search.html#xgboost-hyperparameters
(Of course, it ought to be telling you that, and not crashing, so definitely a bug!)
来源：https://stackoverflow.com/questions/45726324/xgboost-h2o-crashed-due-to-an-illegal-memory-access