i still couldnt finish the db task because i stucked at db creation task intricacies without spark, in which the data types would be incorrect when created with pandas. so I then later swtiched to decided to do create tables with spark type logic. and would continue writing http api for game db on today seems as the db logic has alot intricacies.
so i think in cloud maintenance stuff db creation then would be accomplished with spark (I mean if gcp bucket system wuold be have to reinstalled would be done with a kubernetes script with spark on kubernetes alike. although i had not done like that this time and installed manually from spark task on computer with gcp bucket access settings configured.
so many hours went to this table data type issues tasks.. eg. with pandas i couldnt setup Array<string> type it always created as string. or else all integer types turned out integer64 type. or else time stamp has had incorrect type. so whatever i tried i couldnt fixed these any. so then i decided to switch to spark for db creation task. but would continue to utilize delta lake without spark for game db http api tasks alike.
hmm so initial tests for deltalake libraries without spark, i had issue on here but maybe its because i had not tested all alternate libraries there maybe with some config there its possible to alleviate this issue. dont know.
so today i would first complete these game db tasks which seems important how its coded since its online.
Yorumlar
Yorum Gönder