Packages

c

com.dataforgelabs.sdk

PostOutputSession

class PostOutputSession extends App with PostOutput

Establishes a process record for the custom post output process. Also starts a heartbeat that allows DataForge to track the health of the process. When running manually, must use one of the below constructors. Otherwise, no parameters are necessary.

Linear Supertypes
PostOutput, Util, App, DelayedInit, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PostOutputSession
  2. PostOutput
  3. Util
  4. App
  5. DelayedInit
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PostOutputSession(inOutputName: String, inOutputSourceName: String, inProjectName: String)
  2. new PostOutputSession(inOutputName: String, inOutputSourceName: String)
  3. new PostOutputSession()
  4. new PostOutputSession(inEnvironment: String, inOutputName: String, inOutputSourceName: String, inProjectName: String)

    inEnvironment

    The DataForge environment this ingestion should be run against. Not required, deprecated in release 2.5

    inOutputName

    The DataForge output name this post output should be run against. Mandatory when running manually.

    inOutputSourceName

    The DataForge output source name this post output should be run against. Optional when running manually.

    inProjectName

    The DataForge project name this ingestion should be run against. If not provided, Default project will be used.

Type Members

  1. case class NormalizedAttribute(rawAttributeName: String, columnAlias: String, hiveDdlType: String, upcastExpr: Option[String]) extends Product with Serializable
    Definition Classes
    Util

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. implicit val AttributeSchemaReads: Reads[NormalizedAttribute]
    Definition Classes
    Util
  5. lazy val alertParameters: JsValue
    Attributes
    protected
    Definition Classes
    PostOutput
  6. def args: Array[String]
    Attributes
    protected
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "args should not be overridden" , "2.11.0" )
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def close: Unit
    Attributes
    protected
    Definition Classes
    Util
  10. lazy val cloud: String
    Attributes
    protected
    Definition Classes
    Util
  11. implicit val config: Aux[MacroOptions]
    Definition Classes
    Util
  12. lazy val connectionId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  13. def connectionParameters: JsObject

    Retrieves secure connection parameters from the Databricks Secret store, including public and encrypted private credentials.

    Retrieves secure connection parameters from the Databricks Secret store, including public and encrypted private credentials.

    returns

    JSON object containing connection parameters

  14. def convertToObject(j: JsValue): Object
    Attributes
    protected
    Definition Classes
    Util
  15. def coreApiCall(route: String, verb: String = "GET", body: Option[JsValue] = None): JsValue
    Attributes
    protected
    Definition Classes
    Util
  16. val coreUri: String
    Attributes
    protected
    Definition Classes
    Util
  17. lazy val customParameters: JsObject

    Custom parameters for the post output, supplied via constructor or source configuration.

    Custom parameters for the post output, supplied via constructor or source configuration.

    returns

    JSON object of custom post output parameters

  18. def endProcess(parameters: JsObject): Unit
    Attributes
    protected
    Definition Classes
    Util
  19. lazy val environment: String
    Attributes
    protected
    Definition Classes
    Util
  20. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  22. val executionStart: Long
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( ... , "2.11.0" )
  23. def fail(message: String): Unit

    Fails the post output process, marking the process as failed and writing a failure log.

    Fails the post output process, marking the process as failed and writing a failure log.

    message

    The log message to write along with the failure.

  24. lazy val filePath: String

    Full file path for the post output; applies to file-based outputs.

    Full file path for the post output; applies to file-based outputs.

    returns

    String path to the output file, or null if not applicable

  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  27. def getConnectionParameters: JsObject
    Attributes
    protected
    Definition Classes
    Util
  28. def getProcessSourceName(processId: Int): String
    Attributes
    protected
    Definition Classes
    Util
  29. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  30. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  31. lazy val jobRunId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  32. def log(message: String): Unit

    Writes a log for the process.

    Writes a log for the process. The logs will be visible in the DataForge UI. Severity will default to "I"

    message

    The log message to record.

  33. def log(message: String, severity: String): Unit

    Writes a log for the process.

    Writes a log for the process. The logs will be visible in the DataForge UI.

    message

    The log message to record.

    severity

    1 character to describe severity. Typically "I" for info or "E" for error.

  34. def logFail(e: Exception): Unit
    Attributes
    protected
    Definition Classes
    Util
  35. lazy val logId: Int
    Attributes
    protected
    Definition Classes
    PostOutput → Util
  36. val logger: Logger
    Attributes
    protected
    Definition Classes
    Util
  37. def main(args: Array[String]): Unit
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "main should not be overridden" , "2.11.0" )
  38. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  40. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. lazy val operationType: String
    Attributes
    protected
    Definition Classes
    PostOutput
  42. lazy val originalProcessId: Option[Int]
    Attributes
    protected
    Definition Classes
    Util
  43. lazy val outputSourceId: Int
    Attributes
    protected
    Definition Classes
    PostOutput
  44. val packageName: String
    Definition Classes
    Util
  45. lazy val pgConnectionLog: Connection
    Attributes
    protected
    Definition Classes
    Util
  46. implicit lazy val pgConnectionMain: Connection
    Attributes
    protected
    Definition Classes
    Util
  47. lazy val pgConnectionString: String
    Attributes
    protected
    Definition Classes
    Util
  48. val pgConnectionStringRead: String
    Attributes
    protected
    Definition Classes
    Util
  49. lazy val poParameters: JsObject

    Initial parameters for the post output process, including output name, output source name, process ID, version, package, and optional project name.

    Initial parameters for the post output process, including output name, output source name, process ID, version, package, and optional project name.

    returns

    JSON object of post output initiation parameters

  50. lazy val process: JsValue

    Starts a new post output process in the DataForge backend with the specified parameters.

    Starts a new post output process in the DataForge backend with the specified parameters.

    returns

    JSON value containing process details and output parameters

    Attributes
    protected
    Definition Classes
    PostOutputSession → Util
  51. lazy val processId: Int
    Attributes
    protected
    Definition Classes
    PostOutput → Util
  52. lazy val processParameters: JsObject

    Parameters for the post output process, including file path and custom parameters.

    Parameters for the post output process, including file path and custom parameters.

    returns

    JSON object of process parameters

  53. def run(postOutputCommands: () ⇒ Unit): Unit

    Performs the heavy lifting of the custom post output process, executing user code and handling completion.

    Performs the heavy lifting of the custom post output process, executing user code and handling completion.

    postOutputCommands

    A function that takes no parameters and performs post output work

  54. def runPostOutput(postOutputCommands: () ⇒ Unit): Unit
    Attributes
    protected
    Definition Classes
    PostOutput
  55. lazy val smtpSenderDomain: String
    Attributes
    protected
    Definition Classes
    Util
  56. val spark: SparkSession
    Attributes
    protected
    Definition Classes
    Util
  57. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  58. val systemConfiguration: JsValue
    Attributes
    protected
    Definition Classes
    Util
  59. def toString(): String
    Definition Classes
    AnyRef → Any
  60. val version: String
    Definition Classes
    Util
  61. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  62. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  63. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  64. def writeLog(message: String, severity: String): Unit
    Attributes
    protected
    Definition Classes
    Util
  65. def writeParsedData(inDf: DataFrame, sourceId: Int, inputId: Int, destFilePath: String, forceCaseInsensitiveFlag: Boolean): (Long, Long)
    Attributes
    protected
    Definition Classes
    Util

Deprecated Value Members

  1. def delayedInit(body: ⇒ Unit): Unit
    Definition Classes
    App → DelayedInit
    Annotations
    @deprecated
    Deprecated

    (Since version 2.11.0) the delayedInit mechanism will disappear

Inherited from PostOutput

Inherited from Util

Inherited from App

Inherited from DelayedInit

Inherited from AnyRef

Inherited from Any

Ungrouped