u-sql

问题 I have custom extractor, and I'm trying to log some messages from it. I've tried obvious things like Console.WriteLine , but cannot find where output is. However, I found some system logs in adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/ . How can I log something? Is it possible to specify log file somewhere on Data Lake Store or Blob Storage Account? 回答1: A recent release of U-SQL has added diagnostic logging for UDOs. See the release notes here. // Enable

问题 I'm trying to build CI/CD for Azure Data lake analytics - USQL code and when i build the code using Visual studio build option in VSTS getting the below error - Using the Private agent for taking the build - C:\Users\a.sivananthan\AppData\Roaming\Microsoft\DataLake\MsBuild\1.0\Usql.targets(33,5): Error MSB4062: The "Microsoft.Cosmos.ScopeStudio.VsExtension.CompilerTask.USqlCompilerTask" task could not be loaded from the assembly Microsoft.Cosmos.ScopeStudio.VsExtension.CompilerTask. Could not

问题 I have the Data in Azure Data Lake Store and I am processing the data present there with Azure Data Analytic Job with U-SQL. I have several CSV files which contain spatial data, similar to this: File_20170301.csv longtitude| lattitude | date | hour | value1 ----------+-----------+--------------+------+------- 45.121 | 21.123 | 2017-03-01 | 01 | 20 45.121 | 21.123 | 2017-03-01 | 02 | 10 45.121 | 21.123 | 2017-03-01 | 03 | 50 48.121 | 35.123 | 2017-03-01 | 01 | 60 48.121 | 35.123 | 2017-03-01 |

问题 I need unique guid for every row i'm transforming from source. below is sample script; code Guid.NewGuid() returns same always for all rows @Person = EXTRACT SourceId int, AreaCode string, AreaDetail string, City string FROM "/Staging/Person" USING Extractors.Tsv(nullEscape:"#NULL#"); @rs1 = SELECT Guid.NewGuid() AS PersonId, AreaCode, AreaDetail, City FROM @Person; OUTPUT @rs1 TO "/Datamart/DimUser.tsv" USING Outputters.Tsv(quoting:false, dateTimeFormat:null); 回答1: A quick summary of the

问题 I'm trying to write the results of my custom IOutputter to an intermediate file on the local disk. After that I want to copy the database file (~20MB) to the adl output store. Sadly the script terminates with: An unhandled exception of type 'Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.ScopeDebugException' occurred in Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.dll Additional information: {"diagnosticCode":195887112,"severity":"Error","component":"RUNTIME","source":"User",

问题 I need to get a list of files to then filter this set DECLARE @input_file string = @"\data\{*}\{*}\{*}.avro"; @filenames = SELECT filename FROM @input_file; @filtered = SELECT filename FROM @filenames WHERE {condition} Something like this if it's possible... 回答1: The way to do that is define virtual columns in your fileset. You can then extract and manipulate these virtual columns like they were data fields extracted from your file. Example: DECLARE @input_file string = "/data/{_partition1}/{

问题 I have a JSON where the order of fields is not fixed. i.e. I can have [A, B, C] or [B, C, A] All A, B, C are json objects are of the form {Name: x, Value:y}. So, when I use USQL to extract the JSON (I don't know their order) and put it into a CSV (for which I will need column name): @output = SELECT A["Value"] ?? "0" AS CAST ### (("System_" + A["Name"]) AS STRING), B["Value"] ?? "0" AS "System_" + B["Name"], System_da So, I am trying to put column name as the "Name" field in the JSON. But am

问题 I have an issue with installing the below R-packages and reference them in an R-script I have encapsulated in a U-SQL-script. I succeeded in running a simple R-script in a U-SQL-job that required no special packages. Now I am trying to create an R-script that references dplyr, tdyr and reshape2. Therefore I have downloaded these three packages manually as both .zip and .tar.gz-files and uploaded them to my ADL-account. Example: ../usqlext/samples/R/dplyr_0.7.7.zip The U-SQL startes like this:

问题 I want to extract the filename string as a DateTime column. The code for it as follows: @data = EXTRACT ... filename_date DateTime FROM "/input/vga_{filename_date}.txt" USING Extractors.Tsv(skipFirstNRows:1); filename = vga_20171201.txt whenever i have used datatype as string or int, it's work for me. 回答1: You have to specify .net date format strings along with the virtual column name to get that behaviour, like this: @data = EXTRACT someData string, filename_date DateTime FROM "/input/vga_

问题 I am processing data in U-SQL but not getting expected results. Here is what I am doing: 1- Select data from ADL table partitions and assign it to @data1 2- Aggregate data using Group BY and assign it to @data2 3- Truncate partitions 4- Insert data(produced in step 2) into the same table 5- Use @data2 and generate a unique GUID for every record using user defined function and assign it to @data2 //UDF Code public static Guid GetNewGuid () { return Guid.NewGuid (); } 6- Select few columns from