Packages

c

com.dataforgelabs.sdk

ParsingSession

class ParsingSession extends App with CustomParse

Establishes a process record for the custom parsing process. Also starts a heartbeat that allows DataForge to track the health of the process. When running manually, must use one of the below constructors. Otherwise, no parameters are necessary.

Linear Supertypes
CustomParse, Util, App, DelayedInit, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParsingSession
  2. CustomParse
  3. Util
  4. App
  5. DelayedInit
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ParsingSession(inInputId: Int)
  2. new ParsingSession()
  3. new ParsingSession(inEnvironment: String, inInputId: Int)

    inEnvironment

    The DataForge environment this ingestion should be run against. Not required, deprecated in release 2.5

    inInputId

    The DataForge input ID that will be parsed

Type Members

  1. case class NormalizedAttribute(rawAttributeName: String, columnAlias: String, hiveDdlType: String, upcastExpr: Option[String]) extends Product with Serializable
    Definition Classes
    Util

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. implicit val AttributeSchemaReads: Reads[NormalizedAttribute]
    Definition Classes
    Util
  5. lazy val alertParameters: JsValue
    Attributes
    protected
    Definition Classes
    CustomParse
  6. def args: Array[String]
    Attributes
    protected
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "args should not be overridden" , "2.11.0" )
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def close: Unit
    Attributes
    protected
    Definition Classes
    Util
  10. lazy val cloud: String
    Attributes
    protected
    Definition Classes
    Util
  11. implicit val config: Aux[MacroOptions]
    Definition Classes
    Util
  12. lazy val connectionId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  13. def connectionParameters: JsObject

    Retrieves secure connection parameters from the Databricks Secret store, including public and encrypted private credentials.

    Retrieves secure connection parameters from the Databricks Secret store, including public and encrypted private credentials.

    returns

    JSON object containing connection parameters

  14. def convertToObject(j: JsValue): Object
    Attributes
    protected
    Definition Classes
    Util
  15. def coreApiCall(route: String, verb: String = "GET", body: Option[JsValue] = None): JsValue
    Attributes
    protected
    Definition Classes
    Util
  16. val coreUri: String
    Attributes
    protected
    Definition Classes
    Util
  17. lazy val customParameters: JsObject

    Custom parameters for parsing, supplied via the source configuration.

    Custom parameters for parsing, supplied via the source configuration.

    returns

    JSON object mapping custom parsing parameters

    Definition Classes
    ParsingSession → CustomParse
  18. lazy val datalakePath: String
    Attributes
    protected
    Definition Classes
    CustomParse
  19. lazy val destFilePath: String
    Attributes
    protected
    Definition Classes
    CustomParse
  20. def endProcess(parameters: JsObject): Unit
    Attributes
    protected
    Definition Classes
    Util
  21. lazy val environment: String
    Attributes
    protected
    Definition Classes
    Util
  22. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  24. val executionStart: Long
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( ... , "2.11.0" )
  25. def fail(message: String): Unit

    Fails the parsing process, marking the process as failed and writing a failure log.

    Fails the parsing process, marking the process as failed and writing a failure log.

    message

    The log message to write along with the failure

  26. lazy val filePath: String

    Full path of the file to be parsed, constructed from the datalake path, source ID, and input ID.

    Full path of the file to be parsed, constructed from the datalake path, source ID, and input ID.

    returns

    String path to the source file in the DataForge datalake

  27. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  28. lazy val forceCaseInsensitiveFlag: Boolean
    Attributes
    protected
    Definition Classes
    CustomParse
  29. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  30. def getConnectionParameters: JsObject
    Attributes
    protected
    Definition Classes
    Util
  31. def getProcessSourceName(processId: Int): String
    Attributes
    protected
    Definition Classes
    Util
  32. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  33. lazy val inputId: Int
    Attributes
    protected
    Definition Classes
    CustomParse
  34. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  35. lazy val jobRunId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  36. def log(message: String): Unit

    Writes a log for the process.

    Writes a log for the process. The logs will be visible in the DataForge UI. Severity will default to "I"

    message

    The log message to record.

  37. def log(message: String, severity: String): Unit

    Writes a log for the process.

    Writes a log for the process. The logs will be visible in the DataForge UI.

    message

    The log message to record.

    severity

    1 character to describe severity. Typically "I" for info or "E" for error.

  38. def logFail(e: Exception): Unit
    Attributes
    protected
    Definition Classes
    Util
  39. lazy val logId: Int
    Attributes
    protected
    Definition Classes
    CustomParse → Util
  40. val logger: Logger
    Attributes
    protected
    Definition Classes
    Util
  41. def main(args: Array[String]): Unit
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "main should not be overridden" , "2.11.0" )
  42. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  43. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  44. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  45. lazy val operationType: String
    Attributes
    protected
    Definition Classes
    CustomParse
  46. lazy val originalProcessId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  47. val packageName: String
    Definition Classes
    Util
  48. lazy val pgConnectionLog: Connection
    Attributes
    protected
    Definition Classes
    Util
  49. implicit lazy val pgConnectionMain: Connection
    Attributes
    protected
    Definition Classes
    Util
  50. lazy val pgConnectionString: String
    Attributes
    protected
    Definition Classes
    Util
  51. val pgConnectionStringRead: String
    Attributes
    protected
    Definition Classes
    Util
  52. lazy val process: JsValue

    Starts a new parsing process in the DataForge backend for the specified input ID.

    Starts a new parsing process in the DataForge backend for the specified input ID.

    returns

    JSON value containing process details and parsing parameters

    Attributes
    protected
    Definition Classes
    ParsingSession → CustomParse → Util
  53. lazy val processId: Int
    Attributes
    protected
    Definition Classes
    CustomParse → Util
  54. lazy val processParameters: JsObject

    Parameters for the parsing process, including any standard configuration values.

    Parameters for the parsing process, including any standard configuration values.

    returns

    JSON object of process parameters

  55. def run(parseCommand: () ⇒ DataFrame): Unit

    Performs the heavy lifting of the custom parsing process, executing user code and handling completion.

    Performs the heavy lifting of the custom parsing process, executing user code and handling completion.

    parseCommand

    A function that takes no parameters and returns a DataFrame; this will be executed and saved

  56. def runParse(parseData: () ⇒ DataFrame): Unit
    Attributes
    protected
    Definition Classes
    CustomParse
  57. lazy val smtpSenderDomain: String
    Attributes
    protected
    Definition Classes
    Util
  58. lazy val sourceId: Int
    Attributes
    protected
    Definition Classes
    CustomParse
  59. val spark: SparkSession
    Attributes
    protected
    Definition Classes
    Util
  60. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  61. val systemConfiguration: JsValue
    Attributes
    protected
    Definition Classes
    Util
  62. def toString(): String
    Definition Classes
    AnyRef → Any
  63. val version: String
    Definition Classes
    Util
  64. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  65. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  66. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  67. def writeLog(message: String, severity: String): Unit
    Attributes
    protected
    Definition Classes
    Util
  68. def writeParsedData(inDf: DataFrame, sourceId: Int, inputId: Int, destFilePath: String, forceCaseInsensitiveFlag: Boolean): (Long, Long)
    Attributes
    protected
    Definition Classes
    Util

Deprecated Value Members

  1. def delayedInit(body: ⇒ Unit): Unit
    Definition Classes
    App → DelayedInit
    Annotations
    @deprecated
    Deprecated

    (Since version 2.11.0) the delayedInit mechanism will disappear

Inherited from CustomParse

Inherited from Util

Inherited from App

Inherited from DelayedInit

Inherited from AnyRef

Inherited from Any

Ungrouped