I have to partition the table in hive
with a column which is also part of the table.
For eg:
Table: employee
Co
Maybe, I think that, you should firstly load all the data into one table, then use Hive extension (multiple inserts):
FROM from_statement
INSERT OVERWRITE TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...) [IF NOT EXISTS]] select_statement1
[INSERT OVERWRITE TABLE tablename2 [PARTITION ... [IF NOT EXISTS]] select_statement2]
[INSERT INTO TABLE tablename2 [PARTITION ...] select_statement2] ...;
FROM from_statement
INSERT INTO TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...)] select_statement1
[INSERT INTO TABLE tablename2 [PARTITION ...] select_statement2]
[INSERT OVERWRITE TABLE tablename2 [PARTITION ... [IF NOT EXISTS]] select_statement2] ...;
Then, if you want, you can
from big_data_table
insert overwrite table table1 partiton (ds=2000)
select * where employeeId>0 && employeeId<101>
insert overwrite table table2 partition (ds=4000)
select * where employeeId>=101&&employeeId<=600